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Message  from  the 
Director 


The  High  Performance  Computing  Modernization  Program  (HPCMP)  continues  to  deliver  to  the 
Department  of  Defense  (DoD)  one  of  the  world's  top  supercomputing  infrastructures.  By  the  end  of  2006, 
the  program  had  eight  of  the  world’s  largest  supercomputers  deployed  at  four  Major  Shared  Resource 
Centers  (MSRCs).  (In  terms  of  raw  computing  capacity  the  Program  is  the  second  largest  acknowledged 
program  in  the  world.)  Today,  the  Department  of  Defense  has  access  to  some  of  the  world's  most  powerful 
supercomputers  and  to  a  variety  of  computing  architecmres,  chosen  to  best  meet  the  Department’s  identified 
requirements.  This  allows  our  scientists  and  engineers  to  match  software  applications  to  supercomputers. 
Our  Defense  Research  and  Engineering  Network  provides  connections  to  over  150  sites  with  connection 
speeds  ranging  from  45  to  2,400  megabits  per  second. 

Perhaps,  most  importantly,  our  program  provides  key  computer  and  computational  science  expertise  to 
scientists  and  engineers  across  the  Department.  We  successfully  completed  several  software  development 
projects  that  introduced  parallel,  scalable  production  software  now  in  use  across  the  Department  and 
the  broader  national  community.  This  past  year,  we  delivered  55  training  events,  attended  by  802  people 
and  coordinated  technology  sharing  projects  between  the  defense  laboratories  and  over  two  dozen 
universities. 

These  activities  and  the  daily  work  performed  by  members  of  the  HPCMP  community  have  a  positive 
impact  on  our  national  defense  posture.  Our  scientists  and  engineers  are  now  in  possession  of  tools  once 
only  imagined.  They  are  developing  and  deploying  weather  and  ocean  models  that  allow  our  soldiers, 
sailors,  marines  and  airmen  to  plan  missions  more  effectively  and  to  navigate  adverse  environments  safely. 
They  are  modeling  molecular  interactions  leading  to  the  development  of  higher  energy  fuels,  munitions, 
and  materials  enabling  cheaper,  more  environmentally  friendly  access  to  space,  more  effective  weapons,  and 
stronger  and  longer  lasting  materials.  They  are  modeling  structural  responses  to  different  blast  environments, 
guiding  improved  force  protection  programs,  and  designing  new  guidelines  for  buildings  and  structures. 
Today,  we  support  over  550  individual  projects— research,  advanced  development  and  applied  engineering, 
test  and  evaluation— each  contributing  to  our  national  defense  posture. 

As  the  program  completes  another  year,  we  continue  to  improve  the  Department’s  supercomputing 
environment  supported  by  leading  edge  networking  capabilities.  The  DoD  HPCMP  community  is  working 
in  concert  with  other  federal  agencies  to  identify  future  trends  and  requirements. 

While  the  state  of  commercial  computing  hardware  continues  to  advance  at  a  rapid  pace,  driven  by  a 
huge  commercial  market  and  Defense  Advanced  Research  Projects  Agency  (DARPA)  investments,  advances 
in  parallel  software  driven  primarily  by  federal  investment  have  not  kept  pace  with  the  hardware.  We 
are  planning  an  initiative  to  focus  on  high  performance  software  (which  we  call  Computational  Research 


and  Engineering  Acquisition  Tools  and  Environments  —  CREATE),  to  take  advantage  of  DARPA’s  High 
Productivity  Computing  Systems  (HPCS)  program.  The  budget  for  CREATE  was  included  in  the  President's 
2008  Budget. 

CREATE  will  begin  to  deliver  enhanced  engineering  design  tools  within  3  to  4years  after  the  projects 
begin.  Eully  mature  tools  will  be  delivered  at  the  end  of  a  12-year  project  schedule. 

The  community  is  now  developing  a  plan  for  realizing  the  CREATE  program.  That  plan  is  based  on  the 
"lessons  learned”  from  similar  scale  projects  carried  out  by  the  nuclear  weapons,  the  climate  modelling,  and 
other  computational  communities.  It  builds  on  the  DoD  experience  with  our  institutes  and  portfolios  as 
well.  The  CREATE  12-year  budget  is  about  $350M  with  a  70%  contribution  from  Office  of  the  Secretary  of 
Defense  and  a  30%  matching  contribution  from  the  services.  The  tri-service  T&E  communities  will  provide 
validation  experiments  and  data  for  the  projects  as  well. 

Our  Department's  needs  for  science  and  engineering  to  speed  its  Transformation  Goals  continue  to 
accelerate.  The  High  Performance  Computing  Modernization  Program  team  is  dedicated  to  deploying  and 
operating  superior  supercomputing  environments  and  productivity  enhancing  services  allowing  DoD's 
scientists  and  engineers  to  develop  the  best  technological  solutions  for  our  nation's  defense.  As  President 
Bush  has  said,  "science  and  technology  have  never  been  more  essential  to  the  defense  of  the  nation..." 


CrayJ.  Henry 
Director 

High  Performance  Computing  Modernization  Program 
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Section  1 

Overview 

Section  1  —  HPCMF  Overview 


Section  1 
HPCMP  Overview 


Introduction 

Today,  the  Department  of  Defense  [DoD]  faces 
many  challenges.  We  must  defend  America  by 
maintaining  a  military  second  to  none.  And  we 
must  do  so  affordably. 

The  High  Performance  Computing 
Modernization  Program  [HPCMP]  provides  some 
of  the  tools  the  Department  needs  to  address 
defense  problems.  These  tools  include  modern 
high  performance  computing  hardware  and 
software  and  the  expertise  to  use  them. 

Our  military  strength  depends  on  many 
factors.  Our  people  are  our  greatest  asset  for 
they  are  our  intellectual  capital.  They  include  active 
service  members,  Reserves,  National  Guard,  civil 
servants,  political  appointees,  and  contractors. 
High  performance  computing  [HPC]  hardware 
and  software  help  our  people  make  our  military 
the  best  in  the  world. 

Many  military  problems  are  complicated 
and  often  require  very  powerful  tools  to  be 
solved.  Some  problems  are  too  expensive  for 
experiment  to  address.  Others  are  too  difficult  to 
be  solved  with  paper  and  pencil. 

The  Department  uses  high 
performance  computing  tools  to 
help  solve  some  of  these  hard 
problems.  These  hardware  and 
software  tools  give  us  advantage 
over  potential  adversaries  that 
don’t  have  them. 


Many  modern  weapons  systems  present  hard 
problems.  Early  in  system  development,  we  must 
make  trade-offs  to  balance  performance,  time  and 
available  resources.  How  do  we  determine  cost, 
schedule  and  performance?  How  do  we  take  into 
account  technical  and  management  risks?  HPC 
hardware  and  software  contribute  to  answering 
these  questions. 

HPC  hardware  and  software  help  us  answer 
other  important  questions  as  well.  They  can 
be  used  to  address  a  wide  spectrum  of  issues, 
including:  protecting  our  bases  of  operations 
through  the  mitigation  of  toxic  threats;  modeling 
to  support  certification  of  new  aircraft-store 
combinations  before  deployment  to  conflicts  in 
Afghanistan  and  Iraq;  supporting  US  supremacy 
in  space;  conducting  climate,  weather,  and  ocean 
modeling  that  provides  valuable  information  for 
countermine  warfare  operations;  preparation  for 
emergency  operations  and  humanitarian  relief 
operations  throughout  the  world.  These  are  but 
a  few  examples. 

Over  a  decade,  the  HPCMP  has  supported  a 
workforce  that  routinely  uses  HPC  resources  to 
solve  many  of  the  Department’s  most  challenging 


HPCMP  Mission 

Accelerate  development  and  transition  of  advanced 
Defense  technologies  into  superior  war  fighting 
capabilities  by  exploiting  and  strengthening  US 
leadership  in  supercomputing,  communications,  and 
computational  modeling. 
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HPCMP  Vision 

A  pervasive  culture  existiug  aruoug 
DoD's  scieutists  aud  eugiueers 
where  they  routiuely  use  advauced 
coruputatioual  euvirourueuts  to  solve 
the  ruost  deruaudiug  problerus. 


scientific  and  engineering  problenns.  This,  in  turn, 
helps  the  United  States  ensure  nnilitary  advantage 
and  war-fighting  superiority  on  the  21^^  century 
battlefield. 

The  Progrann  enables  scientists  and  engineers 
to  further  the  Departnnent  s  objectives  through 
research,  developnnent,  test,  and  evaluation 
[RDT&E]  activities  that  support  science  and 
technology  [S&T],  and  test  and  evaluation  [T&E]. 
These  endeavors  focus  on  the  nnost  connplex, 
and  highest  priority  defense  challenges.  This 
annual  report  highlights  a  snnall  portion  of  the 
work  being  done  to  support  the  Departnnent  s 
Transfornnation  Goals.  Defense  scientists  and 
engineers,  using  resources  provided  by  the 
HPCMP,  address  nnulti-disciplinary  scientific  and 
engineering  challenges.  These  include  problenns 
of  interest  across  the  services  and  to  joint 
force  connnnanders.  Improving  the  accuracy  of 
ocean  and  weather  prediction  models,  designing 
materials  for  specific  purposes  such  as  body 
armor  or  agile  laser  eye  protection,  and  modeling 
complex  flow  fields  around  air  systems  to  increase 
performance  are  examples.  Today’s  work  will: 

•  improve  detection  of  targets  based  on  their 
spectral  or  spatial/spectral  signatures; 

•  advance  dynamic  signal  intelligence  mission 
planning; 

•  enhance  force  protection  against  terrorist 
threats;  and 


•  address  the  critical  need  to  develop  new  high 
energy  density  materials  for  explosives  and 
rocket  fuels. 

Congressional  investments  in  and  support  of 
the  HPCMP  since  fiscal  year  1994  have  caused 
cultural  changes  in  the  fundamental  way  S&T 
and  T&E  are  pursued.  In  1993,  the  Department 
had  just  over  ISO  gigaFLOPS  [10^  Floating-point 
operations  per  Second]  [or  GF]  of  computational 
power  to  support  the  S&T  community.  By  the  end 
of  2006,  the  program  had  over  315.5  teraFLOPS 
[10^  gigaFLOPS]  of  computing  capacity,  a  factor 
of  over  1,750  improvement! 

Figure  1  illustrates  the  growth  in  computational 
capabilities  just  at  our  High  Performance 
Computing  Resource  Centers.  This  vast  increase 
in  capability  was  obtained  by  applying  sound 
management  practices  and  good  investment 
strategies.  Similarly,  we  transitioned  our 
communications  network  linking  the  laboratories 
from  a  government-owned,  government-operated 
asset  to  a  commercial  environment  with  secure, 
high  bandwidth  capability. 

The  HPCMP  achieves  the  Program’s  mission 
and  vision  [as  described  on  pages  5  and  6]  by 
focusing  on  five  specific  goals.  Each  activity  within 
the  program  supports  one  or  more  of  these  goals, 
with  progress  tracked  and  successes  delineated. 
These  five  goals  are: 

•  acquire,  deploy,  operate  and  maintain  best- 
value  supercomputers; 

•  acquire,  develop,  deploy  and  support 
software  applications  and  computational 
work  environments  that  enable  critical  DoD 
research,  development  and  test  challenges 
to  be  analyzed  and  solved; 

•  acquire,  deploy,  operate  and  maintain  a 
communications  network  that  enables  effective 
access  to  supercomputers  and  to  distributed 
S&T/T&E  computing  environments; 
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Figure  1.  Computational  capability  at  the  Major  Shared  Resource  Centers  at  the  beginning  of  the  fiscal  year 


•  pronnote  collaborative  relationships  annong 
the  DoD  connputational  science  connnnunity, 
the  national  connputational  science  connnnunity 
and  minority  serving  institutions:  and 

•  continuously  educate  the  research, 
development,  test,  and  evaluation  workforce 
with  the  knowledge  needed  to  employ 
computational  modeling  effectively  and 
efficiently. 

The  progress  the  HPCMP  has  made  in 
meeting  these  goals  is  discussed  in  detail  later 
in  Section  2. 

The  HPCMP  Community 

The  HPCMP  community  consists  of  nearly 
4,000  scientists,  engineers,  computer  specialists. 


networking  specialists,  and  security  experts 
working  throughout  the  United  States.  All  three 
Military  Departments  and  several  Defense 
Agencies  participate  in  the  program.  These  users 
execute  over  500  projects— each  validated  by  the 
Military  Services  and  Defense  Agencies.  Figure  2 
shows  the  locations  of  people  using  the  program  s 
resources.  The  user  base  is  diverse,  drawing 
from  the  government  workforce,  academia,  and 
industry.  The  demographics  by  type  of  workforce 
as  well  as  by  the  DoD  organizations  are  shown 
in  Figure  3.  Most  work,  done  by  the  HPCMP 
community,  is  in  one  or  more  of  the  program  s 
ten  computational  technology  areas  [CTAs]  [see 
Table  1  on  page  10].  Figure  4  includes  a  break¬ 
down  of  users  by  CTA. 
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Figure  2.  The  light  green  color  represents  states  with  people  using  HPCMP  resources 


HPCMP  Community 
Computational  Requirements 

Validated  requirennents  serve  as  the  basis  for 
HPCMP  investnnents  and  operational  decisions. 
Each  yean  the  program  gathers,  assesses,  and 
validates  user  community  requirements.  This 
requirements  determination  includes  all  aspects 
of  HPCMP  activities  and  capabilities:  system 
hardware,  software,  networking,  and  training. 
In  the  past  year,  overall  requirements  increased 
and  are  projected  to  continue  to  increase  at  a 
steady,  consistent  rate.  Total  requirements  in 
any  given  year  are  approximately  two  and  one-half 
times  total  program  capability,  ramping  up  from 
approximately  540  teraFLOPS  or  310  Habus  in 
FY  2007  to  1,752  teraFLOPS  or  2,190  Habus  in 
FY  2011  [see  Figure  5].  [Habus  are  a  measure 
of  computational  performance.  See  callout  on 
page  11  for  a  definition.]  Once  collected,  the 
Services’  and  Agencies’  S&T  and  T&E  Executives 
review,  correct,  validate,  and  approve  their 
requirements.  HPCMP  conducts  requirements 
analyses  as  a  fundamental  part  of  an  overall 


systems  engineering  process  that  collects 
and  analyzes  information  to  make  investment 
decisions. 

The  general  conclusion  of  that  requirements 
analysis  is  that  a  complete  HPC  environment 
must  be  provided  to  support  the  DoD’s  S&T  and 
T&E  communities.  A  spectrum  of  computational 
platforms,  both  at  the  unclassified  and  classified 
levels,  must  be  provided  so  that  a  wide  range  of 
DoD  applications  may  be  efficiently  supported. 
These  platforms  must  be  balanced  with  respect 
to  computational  power,  central  memory,  and 
file  storage  capabilities.  A  variety  of  systems 
and  applications  software  that  enable  DoD 
computational  scientists  and  engineers  to 
perform  their  mission  are  required.  A  reliable 
high-speed  network  that  connects  the  users  to 
these  resources  and  to  each  other  is  required, 
as  is  the  continuation  of  an  aggressive  training 
program  that  broadens  and  educates  DoD’s  HPC 
users.  Progress  must  be  balanced  across  all 
program  activities  to  optimize  the  impact  of  HPC 
on  the  DoD  S&T  and  T&E  programs’  support  of 
the  war  fighting  mission. 
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Categorization  of  DoD  HPC  Users  by 
Service/Agencies 


Categorization  of  DoD  HPC  Users  by 
Government  or  Contractor 


Distribution  of  DoD  HPC  Users  as  Government  or 
Contractor  by  Service/Agency 


Agencies 

Contractor  Army  Gov 


9%  Navy 

Contractor 

12% 


On-Site 

Contractors 

25% 

Figure  3.  FY  2006  HPC  user  demographics 


HPCMP  User  Base 


587  projects  and  4,617  users  at 
approximately  133  sites 

Requirements  categorized  in  10 
computational  technology  areas  (CTAs) 

FY  2006  non-real-time  requirements  of 
291  Habu  equivalents 


Computational  Chemistry,  Biology 
&  Materials  Science  -  330  Users 


Computational  Fluid 
Dynamics  -  1,226  Users 


Integrated  Modeling  &  Test 
Environments  -  490  Users 


Forces  Modeling  & 
Simulation  -780  Users 


Computational  Structural 
-  532  Users 


Environmental  Quality  Modeling 
&  Simualtion  -  183  Users 


Electronics,  Networking,  and 
Systems/C4l-  34  Users 


Signal/lmage  Processing  - 
433  Users 


Climate/Weather/Ocean  Modeling 
&  Simulation  -  780  Users 


Computational  Electromagnetics 
&  Acoustics  -  319  Users 


62  users  are  self  characterized  as  “Other” 


Figure  4.  HPCMP  serves  a  large,  diverse,  Department  of  Defense,  user  community 
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Table  1.  Computational  Technology  Areas  (CTAs) 


Computational  Technology  Area  Acronym  Description 


Computational  Structural  Mechanics  CSM 

Computational  Fluid  Dynamics  CFD 

Computational  Chemistry,  Biology,  and  Materials  CCM 

Science 

Computational  Electromagnetics  and  Acoustics  CEA 

Climate/Weather/Ocean  Modeling  and  CWO 

Simulation 

Signal/lmage  Processing  SIP 

Forces  Modeling  and  Simulation  FMS 

Environmental  Quality  Modeling  and  Simulation  EQM 

Electronics,  Networking,  and  Systems/C4l  ENS 

Integrated  Modeling  and  Test  Environments  IMT 


Covers  the  high  resolution,  multi-dimensional  modeling  of 
materials  and  structures  subjected  to  a  broad  range  of 
loading  conditions  such  as  quasi-static,  dynamic,  electro¬ 
magnetic,  shock,  penetration,  and  blast. 

Provides  accurate  numerical  solution  of  the  equations 
describing  fluid  and  gas  motion. 

Predicts  properties,  and  simulates  the  behavior,  of 
chemicals  and  materials  for  DoD  applications.  Methods 
ranging  from  quantum  mechanical,  atomistic,  and 
mesoscale  modeling,  to  multiscale  theories  that  address 
challenges  of  length-  and  time-scale  integration,  are 
being  developed  and  applied.  Of  recent  emerging 
interest  in  the  CCM  CTA  are  methodologies  that  cover 
bioinformatics  tools,  computational  biology,  and  related 
areas,  such  as  cellular  modeling. 

Provides  high-resolution  multidimensional  solutions  of 
electromagnetic  and  acoustic  wave  propagation,  and 
their  interaction  with  surrounding  media. 

Involves  accurate  numerical  simulation  and  forecast  of 
the  Earth’s  atmosphere  and  oceans  on  those  space  and 
time  scales  important  for  both  scientific  understanding 
and  DoD  operational  use. 

Extracts  and  analyzes  key  information  from  various 
sensor  outputs  in  real-time:  sensor  types  include  sonar, 
radar,  visible  and  infrared  images,  signal  intelligence,  and 
navigation  assets. 

Focuses  on  the  research  and  development  of  HPC-based 
physical,  logical,  and  behavioral  models  and  simulations 
of  battlespace  phenomena  in  the  correlation  offerees. 

Involves  the  high-resolution  modeling  of  hydrodynamics, 
geophysics,  and  multi-constituent  fate/transport 
through  the  coupled  atmospheric/land  surface/ 
subsurface  environment,  and  their  interconnections  with 
numerous  biological  species  and  anthropogenic  activities. 

Focuses  on  the  use  of  computational  science  in 
support  of  analysis,  design,  modeling,  and  simulation 
of  electronics  from  the  most  basic  fundamental,  first 
principles  physical  level  to  its  use  for  communications, 
sensing,  and  information  systems  engineering:  activity 
ranges  from  the  analysis  and  design  of  nano-devices  to 
modeling  systems-of-systems. 

Addresses  the  application  of  integrated  modeling  and 
simulation  tools  and  techniques  with  live  tests  and 
hardware-in-the-loop  simulations  for  the  testing  and 
evaluation  of  DoD  weapon  components,  subsystems, 
and  systems  in  virtual  and  composite  virtual-real 
environments. 
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Figure  5.  Total  computational  requirements  of  the  HPCMP  community 


HPCMP  Organization 

The  HPCMP  is  comprised  of  three  major 
components:  HPC  Centers,  Networking  and 
Information  Assurance,  and  Software  Applications 
Support.  These  components  provide  the  base  of 
the  integrated  program  strategy  [see  Figure  6]  to 
provide  a  technologically  advanced  computational 
environment  to  support  the  ongoing  and  emerging 
needs  of  the  Departments  laboratories  and  test 


Habu— A  Measure 


centers.  These  components  are  interdependent, 
with  distinct  business  practices  and  community 
relationships. 

The  HPC  Centers  component  includes  four 
major  shared  resource  centers  [MSRCs]  and 
four  allocated  distributed  centers  [ADCs].  These 
computer  centers  provide  DoD  scientists  and 
engineers  with  the  resources  necessary  to  solve 
the  most  demanding  computational  problems. 
Additional  computational  resources  are  provided 

OF  Computational 


The  HPCMP  rates  computer  systems  in  terms  of  the  speed  at  which  DoD  computational  applications  run 
on  the  systems.  For  the  past  six  years,  the  HPCMP  has  run  a  suite  of  applications  on  existing  and  new 
systems  to  obtain  performance  comparisons.  By  comparing  the  timing  results  for  these  applications, 
the  HPCMP  is  able  to  compare  the  performance  of  any  system  relative  to  the  others.  In  2002,  a  large 
IBM  system  located  at  the  Naval  Oceanographic  Office  MSRC  named  Habu,  was  designated  the  baseline 
system.  Hence,  performance  measures  are  all  in  ''Habu"  equivalent  units.  For  example,  if  a  new  system 
is  rated  at  2  Habus,  that  system  is  roughly  two  times  more  capable  than  a  system  rated  at  1  Habu.  That 
is,  the  new  system  executes  the  suite  of  applications  at  roughly  twice  the  performance  of  the  old.  Of 
course,  any  individual  application  may  run  faster  or  slower.  The  line  in  Figure  5  shows  the  growth  in 
computational  requirements  in  Habu  units  of  system  performance. 
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Figure  6.  HPCMP  integrated  program  strategy 
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to  support  specific  projects  if  those  projects  can 
not  be  easily  addressed  at  the  HPC  Centers  in  a 
shared  resource  environnnent.  These  resources 
are  ternned  dedicated  HPC  project  investnnents 
[DHPIs].  Figure  7  shows  the  MSRCs,  ADCs  and 
DHPIs. 

The  Networking  and  Infornnation  Assurance 
connponent  of  the  HPCMP  includes  the  Defense 
Research  and  Engineering  Network  [DREN].  DREN 
provides  advanced  connnnunication  capabilities  at 
faster  speeds  to  a  bigger  user  community  than 
previously  possible,  while  addressing  our  security 
requirements. 

The  software  applications  support  [SAS] 
component  provides  expert  services  to  assist 
our  customers  in  most  effectively  using  the  HPC 
systems,  provides  investments  in  human  capital 
across  the  DoD  to  facilitate  the  application  of  HPC 
tools,  and  supports  a  modest  investment  in  a  few 
high  need  HPC  software  applications. 

DoD  Challenge  Projects 

The  HPCMP  supports  high  priority 
computational  work  conducted  within  DoD  that 
can  be  done  at  its  shared  resource  centers 
through  Challenge  Projects.  These  projects 
represent  the  DoD  s  highest-priority,  highest- 
impact  computational  work,  both  from  technical 
and  mission-relevance  standpoints.  The  modeling 
and  simulations  conducted  by  these  projects 
account  for  approximately  35%  of  the  allocations 
of  resources  at  the  HPC  centers.  These  projects 
range  from  discovering  new  materials  using 
quantum  chemical  simulations  to  studying  the 
impact  of  new  physics  in  the  prediction  of  weather. 
There  were  37  active  DoD  Challenge  Projects  in 
FY  2006—23  continuing  projects  and  14  new 
projects.  The  14  new  projects  were  selected 
from  29  proposals  submitted  by  the  Services 
and  Agencies  in  response  to  the  HPCMP  s  annual 
call  for  Challenge  Project  proposals.  Selections 


were  made  by  peer  review  with  a  panel  consisting 
of  service,  agency,  DoD,  and  external  reviewers. 
Table  2  lists  the  FY  2006  DoD  Challenge  Projects. 
Almost  all  Challenge  Project  Leaders  presented 
the  results  of  their  work  at  the  annual  Users 
Group  Conference  held  in  Denver,  CO  in  June 
2006. 

Dedicated  HPC  Project 
Investments  (DHPIs) 

The  HPCMP  also  supports  high  priority 
computational  work  conducted  within  DoD  that 
requires  dedicated  HPC  resources.  These 
projects  typically  have  a  need  for  quick  turnaround 
of  the  computational  work,  either  actual  real-time 
or  near-real-time  calculations  often  in  support  of 
a  specific  test  event.  Such  requirements  are  met 
through  the  HPCMP  s  implementation  of  DHPIs. 
These  small  to  medium-sized  projects  require 
HPC  resources  that  have  one  or  more  of  the 
following  attributes: 

•  require  access  to  data  or  computational 
resources  undertime  critical  constraints  that 
can  not  tolerate  network  latency  or  shared 
computing; 

•  signal  image  processing  real-time  attribute; 

•  embedded  systems  applications; 

•  dedicated  computational  resources  available 
immediately  as  needed; 

•  early  access  technology  evaluation; 

•  require  special  operational  considerations, 
including  security  requirements  or 
unconventional  operations. 

Examples  of  the  types  of  projects  supported 
by  DHPI  resources  include: 

•  real-time  analytic  and  decision  support  in  test 
and  evaluation  of  land  combat  systems; 

•  platform  for  conducting  operational  tests  of 
weather  research  and  forecast  models; 
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Table  2.  FY  2006  DoD  Challenge  Projects 


Project  Title 


Project  Leader/Organization 


Advanced  Chemical  Oxygen-Iodine  Laser  Technology 
Development  Using  3-D  Navier-Stokes  Simulation 

Applications  of  Time-Accurate  CFD  in  Order  to  Account 
for  Blade-Row  Interactions  and  Distortion  Transfer 
in  the  Design  of  High  Performance  Military  Fans  and 
Compressors 

Applied  Computational  Fluid  Dynamics  [ACFSD]  in  Support 
of  Aircraft-Store  Compatibility  and  Weapons  Integration 

Characterization  and  Prediction  of  Stratospheric  Optical 
Turbulence  for  DoD  Directed  Energy  Platforms 

Computational  Simulations  of  Combustion  Chamber 
Dynamics  and  Hypergolic  Gel  Propellant  Chemistry  for 
Selectable  Thrust  Engines  in  Next  Generation  Guided 
Missiles 

Computational  Studies  of  Naval  SONAR  and  NVRAM 
Devices 

Computer  Design  and  Simulation  of  Molecular  Devices  and 
Energy  Sources  for  Naval  Applications 

Coupled  Aircraft/Ship  Performance  Prediction  for 
Dynamic  Interface 

Coupled  CFD/CSM/DPM  Modeling  of  Structure  Response 
to  Blast  Loading 

Design  of  Energetic  Ionic  Liquids 

Design  of  Materials  for  Laser  Protection  Applications 

Distributed  Pump  Jet  Propulsion  [DPJP]  for  Submarines 


Dynamic  Rotorcraft  Simulations  for  Accurate  Interactional 
Aerodynamics  and  Performance  Prediction 

Explosive  Structure  Interaction  Effects  in  Urban  Terrain 

First-Principle  Predictions  of  Crystal  Structure  of  Energetic 
Materials 

Global  Dcean  Prediction  with  HYCDM 

High  Accuracy  DNS  and  LES  of  High  Reynolds  Number, 
Supersonic  Base  Flows  and  Passive  Control  of  the  Near 
Wake 

High  Fidelity  Electromagnetic  Target  Signatures  for  Combat 
Identification 


Timothy  Madden,  Air  Force  Research  Laboratory,  Kirtland 
AFB,  NM 

Steven  E.  Gorrell,  Air  Force  Research  Laboratory,  Wright- 
Patterson  AFB,  DH 


Jacob  Freeman,  Air  Force  SEEK  EAGLE  Dffice,  Eglin  AFB,  FL 

Frank  H.  Ruqqiero,  Air  Force  Research  Laboratory,  Hanscom 
AFB,  MA 

Michael  Nusca  and  Michael  McOuaid,  Army  Research 
Laboratory,  Aberdeen  Proving  Ground,  MD 


Andrew  M.  Rappe,  University  of  Pennsylvania,  Philadelphia,  PA 

Mark  R.  Pederson,  Naval  Research  Laboratory,  Washington, 
DC 

Susan  Polsky,  Naval  Air  Warfare  Center,  Aircraft  Division, 
Patuxent  River,  MD 

Joseph  D.  Baum,  Science  Applications  International 
Corporation,  McLean,  VA  [Defense  Threat  Reduction  Agency] 

Jerry  Boatz,  Air  Force  Research  Laboratory,  Propulsion 
Directorate,  Edwards  AFB,  CA 

Ruth  Pachter,  Air  Force  Research  Laboratory,  Materials  and 
Manufacturing  Directorate,  Wright-Patterson  AFB,  DH 

Joseph  Gorski,  Naval  Surface  Warfare  Center,  Carderock 
Division,  West  Bethesda,  MD,  and  Robert  Kunz,  Pennsylvania 
State  University,  State  College,  PA  [Dffice  of  Naval  Research] 

Mark  Potsdam,  US  Army  Aviation  and  Missile  Command, 
Moffett  Field,  CA 

James  T  Baylot,  Army  Engineer  Research  and  Development 
Center,  Vicksburg,  MS 

Krzysztof  Szalewicz,  University  of  Delaware,  Newark,  DE  [Army 
Research  Dffice] 

Alan  Wallcraft,  Naval  Research  Laboratory,  Stennis  Space 
Center,  MS 

Hermann  Fasel,  University  of  Arizona,  Tucson,  AZ  [Army 
Research  Dffice] 

Mary  Ann  Gualtieri,  Air  Force  Research  Laboratory,  Sensors 
Directorate,  Wright-Patterson  AFB,  DH 
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Table  2.  FY  2005  DoD  Challenge  Projects— continued 


Project  Title 


Project  Leader/Organization 


High  Resolution  Simulation  of  Full  Aircraft  Control  at  Flight 
Reynolds  Numbers 

Hypersonic  Scramjet  Technology  Enhancements  for  Long 
Range  Interceptor  Missile 

Millimeter-Wave  Radar  Signature  Prediction  Improvement 
for  Ground  Vehicles 

Modeling  Breaking  Ship  Waves  for  Design  and  Analysis  of 
Naval  Vessels 

Molecular  Rotors  for  Nanotechnology 

Multidisciplinary  Computational  Terminal  Ballistics  for 
Weapons  Systems 

Multi-Scale  Predictability  of  High-Impact  Weather  in  the 
Battlespace  Environment 

Multiscale  Simulations  of  Nanotubes  and  Quantum 
Structures 

Prediction  Capability  for  High-Speed  Surface  Ships 

Scalable  Multiscale  Simulations  of  Material  Behavior  at  the 
Nanoscale 

Simulation  of  a  Dynamically  Maneuvering  Unmanned 
Combat  Air  Vehicle 

Simulation  of  Enhanced  Explosive  Devices  in  Chambers 

Simulations  for  Microbubble  Drag  Reduction  at  High 
Reynolds  Number 

Solidification  of  Complex  High  Temperature  Structural 
Analysis 

Statistical  Fatigue  and  Residual  Strength  Analysis  of  New 
and  Aging  Aircraft  Structure 

Time-Accurate  Coupled  CFD/RBD  Simulations  of  Free 
Flight  Aerodymanics  of  Guided  Weapons 

Tip-to-Tail  Turbulent  Scramjet  Flowpath  Simulation  with 
MHD  Energy  Bypass 

Toward  a  High-Resolution  Global  Coupled  Navy  Prediction 
System 

Virtual  Prototyping  of  Directed  Energy  Weapons 


Scott  Morton,  US  Air  Force  Academy,  Colorado  Springs,  CD 

Kevin  Kennedy,  US  Army  Aviation  and  Missile  Command, 
Redstone  Arsenal,  AL  and  CRAFT  Tech,  Dublin,  PA 

William  Coburn,  Army  Research  Laboratory,  Aberdeen 
Proving  Ground,  MD 

Dick  K.P  Yue,  Massachusetts  Institute  of  Technology, 
Cambridge,  MA  [Dffice  of  Naval  Research] 

Josef  Michl,  University  of  Colorado,  Boulder,  CD  [Army 
Research  Dffice] 

Kent  Kimsey  and  David  Kleponis,  Army  Research  Laboratory, 
Aberdeen  Proving  Ground,  MD 

James  Doyle,  Naval  Research  Laboratory,  Marine 
Meteorology  Division,  Monterey,  CA 

Jerry  Bernholc,  North  Carolina  State  University,  Raleigh,  NC 
[Dffice  of  Naval  Research] 

Joseph  Gorski,  Naval  Surface  Warfare  Center,  Carderock 
Division,  West  Bethesda,  MD 

Rajiv  K.  Kalia,  Aiichiro  Nakano,  and  Priya  Vashishta,  University 
of  Southern  California,  Los  Angeles,  CA  [Army  Research 
Dffice] 

Raymond  Gordnier,  Air  Force  Research  Laboratory,  Wright- 
Patterson  AFB,  DH 

John  B.  Bell,  Lawrence  Berkeley  National  Laboratory,  Berkeley, 
CA  [Defense  Threat  Reduction  Agency] 

Martin  Maxey,  Brown  University,  Providence,  Rl  [Defense 
Advance  Research  Projects  Agency] 

Christopher  Woodward,  Air  Force  Research  Laboratory, 
Materials  and  Manufacturing  Directorate,  Wriqht-Patterson 
AFB,  DH 

Scott  Fawaz,  US  Air  Force  Academy,  Colorado  Springs,  CD 

Jubaraj  Sahu,  Army  Research  Laboratory,  Aberdeen  Proving 
Ground,  MD 

Datta  Gaitonde,  Air  Force  Research  Laboratory,  Wright- 
Patterson  AFB,  DH 

Julie  McClean,  Naval  Postgraduate  School,  Monterey,  CA 

Keith  Cartwright,  Air  Force  Research  Laboratory,  Kirtland 
AFB,  NM 
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•  real-time  global-scale  computer-generated 
forces  experimentation; 

•  real-time  hardware-in-the-loop  avionics  and 
weapon  systems  simulations  for  test  and 
evaluation; 

•  modeling  and  simulation  of  command,  control, 
communication,  computers,  intelligence, 
surveillance  and  reconnaissance  [C4ISR] 
electronic  systems  under  realistic  tactical 
conditions;  and 

•  real-time  data  imaging  of  aerial  objects. 

The  HPC  systems  are  procured  eitherthrough 
the  annual  technology  insertion  process  [TI-XX] 
that  acquires  new  computational  capability  for 
HPCMP  centers,  or  by  providing  procurement 
funding  for  the  systems  directly  to  the  user  site 
that  proposed  the  dedicated  HPC  project. 

Four  DHPIs  awards  were  made  in  FY  2006; 
these  include  projects  to  be  hosted  at:  [1  ]  Dugway 
Proving  Grounds,  LIT  for  extending  the  existing  four¬ 
dimensional  weather  system  to  support  ensemble 
prediction  techniques  to  provide  probability 
distribution  of  weather  variables  across  the  test 
range,  [2]  the  Army  Communications  Electronics 
Research  and  Development  Engineering 
Center  at  Fort  Monmouth,  NJ  for  use  of  high- 
fidelity  software  models  to  perform  analyses  of 
integrated  C4ISR  technologies  and  phenomena, 
and  [3]  MIT  Lincoln  Laboratory,  Lexington,  MA 
to  expand  parameter  spaces  and  problem  sizes 
that  can  be  exploited  and  translated  into  DoD 
systems  that  are  better  able  to  detect  faint  enemy 
signals,  analyze  intelligence  and  reconnaissance, 
and  intercept  ballistic  missiles.  A  fourth  project 
was  subsequently  terminated  prior  to  resource 
procurement  because  of  a  loss  of  critical  project 
technical  expertise. 

User  organizations  that  were  awarded  HPC 
resources  to  support  their  projects  in  prior  years 
and  reached  milestone  completion  presented 
reports  at  the  annual  Users  Group  Conference 
in  June  2006.  A  typical  DHPI  has  a  life-cycle  of 


two  to  three  years,  depending  on  the  projects 
established  milestones.  In  FY  2006,  six  DHPI 
projects  from  the  following  sites  were  transitioned 
from  HPCMP  oversight:  Joint  Forces  Command, 
Fleet  Numerical  Meteorology  and  Oceanography 
Center/Air  Force  Weather  Agency,  Army 
Technical  Center,  Naval  Surface  Warfare  Center, 
Carderock  Division,  Air  Force  Seek  Eagle  Office, 
Arnold  Engineering  Development  Center,  and  Naval 
Research  Laboratory.  Accordingly,  the  following 
DHPIs  are  currently  under  HPCMP  oversight: 

•  Naval  Undersea  Warfare  Center,  Newport, 
Rl; 

•  Space  and  Naval  Warfare  Systems  Center, 
San  Diego,  San  Diego,  CA; 

•  MIT-Lincoln  Labs,  Bedford,  MA; 

•  Command  Electronics  Research,  Development 
and  Engineering  Center,  Ft.  Monmouth,  NJ; 
and 

•  Dugway  Proving  Grounds,  Dugway,  UT. 

Figure  8  lists  the  DHPIs  for  FY  05,  FY  06, 
and  FY  07. 

Capability  Applications 
Projects  (CAPs) 

Starting  in  FY  2005,  the  HPCMP  made 
available  newly  acquired  systems  for  capability 
applications  projects,  designed  to  test  key  DoD 
application  codes  on  a  substantial  portion  of 
entire  HPC  systems  and  solve  large  problems 
quickly.  Thus,  the  goals  of  capability  applications 
projects  are: 

•  to  quantify  the  degree  to  which  important 
application  codes  scale  to  thousands  of 
processors;  and 

•  to  enable  new  science  and  technology  by 
applying  these  codes  in  dedicated,  high-end, 
capability  environments. 
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FY07 


Dedicated  HPC  Project  investments  (U& 

FY  2005-FY  2007 


Application  of  HPC  to  Support  Operational  Use  of  CT-Analyst 

Naval  Research  Laboratory,  Washington,  DC  and  Navy  Special  Warfare  Command 
Mission  Support  Center,  Coronado,  CA 

Distributed  Continuous  Experimentation  Environment,  Interactive  Joint  Features  Laboratory 
Joint  Forces  Command,  Suffolk,  VA 

Joint  Ensemble  Forecast  System 

Air  Force  Weather  Agency,  Offutt  AFB,  NE  and  Fleet  Numerical  Meteorology,  Oceanography  Center,  Monterey,  CA 
Joint  Warfare  Systems  Server 

Office  of  the  Secretary  of  Defense  (Program  Analysis  &  Evaluation),  Washington,  DC 


FY06 

Interactive  Algorithm  Development  for  WMD  Defense 

Massachusetts  Institute  of  Technology-Lincoln  Laboratory,  Lexington,  MA 

Operational,  Probabilistic,  Numerical  Weather  prediction  at  High  Resolution, 

Dugway  Proving  Ground,  UT 

Virtual  Electronic  Battlefield 

Command  Electronics  Research,  Development  and  Engineering  Center,  Ft.  Monmouth,  NJ 


FY05 

Aircraft  Store  Certification 

Arnold  Engineering  Development  Center,  Arnold  AFB,  TN  and  Air  Force  Seek  Eagle  Office,  Eglin  AFB,  FL 

Concurret  Computation  and  Utilization  Environment 
Naval  Surface  Warfare  Center,  Bethesda,  MD 

Space  Situational  Awareness 

Air  Force  Maui  Optical  &  Supercomputing  Site,  Kihei,  HI 

Torpedo  Hardware-in-the-Loop  Modeling  &  Simulation, 

Naval  Undersea  Warfare  Center,  Newport,  Rl 


Figure  8.  Dedicated  HPC  Project  Investments  for  fiscal  years  05  through  07 


The  process  is  an  extension  of  pioneer  usage 
of  new  HPC  systenns,  but  it  focuses  much  more 
heavily  on  true  capability  use  for  a  short  time 
before  those  systems  are  put  into  allocated 
operational  use.  It  is  implemented  in  two  phases: 
Phase  I,  which  focuses  on  scalability  testing  of 
applications  codes  proposed  for  CAPs,  and  Phase 
II,  during  which  a  subset  of  successfully  tested 
codes  and  projects  have  dedicated  access  to  the 
newly  acquired  systems  for  production  work  to 
solve  a  large,  significant  problem.  The  period  of 
time  dedicated  to  this  capability  workload  typically 
lasts  from  one  to  three  months. 


Because  of  late  delivery  of  TI-06  systems, 
there  were  no  CAPs  executed  in  FY  2006.  FY 
2007  plans  include  the  execution  of  CAPs  on  most 
of  the  TI-06  systems  in  addition  to  several  of  the 
TI-07  systems.  A  report  on  those  projects  will 
be  made  in  the  FY  2007  Annual  Report.  In  the 
interim,  a  vignette  of  one  effort  nearing  completion 
appears  on  page  18. 
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Computational  Research  and 
Engineering  Acquisition  Tools 
AND  Environments  (CREATE) 

Computer  power  has  grown  exponentially 
during  the  last  60  years.  By  2012,  the  DoD 
computational  science  and  engineering  community 
will  have  access  to  computers  with  peak 
processing  speeds  in  the  petaFLOPS  [10^^  FLOPS] 
range.  Computer  power  of  this  magnitude  offers 
the  potential  for  DoD  computational  applications 
to  employ  highly  accurate  numerical  methods, 
include  all  physical  effects  known  to  be  important, 
predict  the  behavior  of  a  complete  system  like  an 
entire  airplane,  and  obtain  results  quickly  enough 
that  users  can  make  parameter  and  sensitivity 


studies.  Such  computer  power  has  the  potential 
to  give  DoD  engineers  the  opportunity  to  produce 
improved  designs  and  detect  and  fix  design  flaws 
before  major  schedule  and  funding  commitments 
have  been  made. 

Realizing  this  potential  will  require  development 
of  new  software  applications  to  incorporate 
complete  sets  of  accurate  models  and  exploit  the 
power  of  these  new,  massively  parallel  and  very 
complex  supercomputers.  In  the  past,  DoD  relied 
heavily  on  application  codes  developed  by  other 
agencies  such  as  the  Department  of  Energy  [DDE], 
National  Aeronautics  and  Space  Administration 
[NASA],  academia,  and  commercial  vendors. 
However,  in  the  future  DoD  will  need  to  develop 
more  of  its  engineering  design  and  analysis  tools. 


Parallel  Algorithms  for 
Computational  Fluid  Dynamios 
Simulations  of  Turbulenoe 


Aircraft  designs  require  an  understanding  of  subsonic  jet  flowS;  modeling  accurately  and  rapidly  the  nozel  geometry, 
flow  structures  produced  by  the  interaction,  and  merger  of  co-rotating  wing-tip  vortices  generated  in  flight  Recently, 
collaborators  at  the  Air  Force  Research  Laboratory  and  the  College  of  William  &  Mary  completed  the  largest  computational 
fluid  dynamics  simulations  of  turbulence  to  date  using  a 
unique  parallel  computing  approach.  They  did  so  on  the 
newest  DoD  supercomputer,  located  at  the  NAVO  MSRC 
at  the  Stennis  Space  Center  in  Mississippi,  through  a  Phase 
1 1  CAP  titled  ''Quantum  and  Entropic  Algorithms  for  MHD 
and  Turbulent  Flows." 


They  tested  new  sub-grid  models  of  turbulence  developed 
by  their  group,  using  lattice  Boltzmann  equation  techniques, 
an  entropic  method  and  a  Smagorinsky  closure  method. 
These  CAP  ll  simulations  took  over  500,000  hours,  used  a 
dedicated  block  of  thousands  of  high  performance  processors 
over  the  period  of  a  few  weeks,  and  generated  terabytes  of 
data  per  day.  These  large-scale  simulations  provide  a  better 
understanding  of  the  morphological  evolution  and  structural 
development  of  turbulence  in  fluids. 


Entropic  27-particle  lattice  Boltzmann  simulation  of  turbulence. 


A  single  job  took  many  days  on  2048  processors  on  the  system. 
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Separated  flow 

^Loss  of  control 


DDG-1000 


Damage  from  full 
ship  shock  test 


C4ISR  and  sensing 
antennas  in  network 
centric  warfare 


Figure  9.  Examples  of  aircraft  design,  ship  design,  and  antenna  design 


The  reasons  for  this  are: 

1 .  other  governnnent  agencies  are  focusing  on 
their  own  nnission  requirennents; 

2.  the  nnarket  niche  is  too  snnall  to  allow  vendors 
to  be  connnnercially  successful:  and 

3.  developing  new  large-scale,  nnassively  parallel 
tools  is  an  expensive,  long-ternn  effort. 

The  recent  Quadrennial  Defense  Review 
[2006,  pp.  4,  67-71  ],  and  the  2006  Government 
Accountability  Office  report  [GA0-0B-585T,  2006] 
highlight  the  need  to  improve  the  DoD  acquisition 
process.  Consequently,  during  FY  2006,  the 
HPCMP  community  proposed  the  CREATE 
program  as  an  initiative  in  the  budget  process. 

Present  acquisition  programs  largely  follow  a 
“build,  test  and  break,  fix,  build, ...”  methodology.  This 
results  in  late  discovery  of  design  flaws,  immature 
technologies  issues,  and  system  integration 
problems,  causing  costly  rework  and  redesign 
that  contribute  substantially  to  cost  overruns  and 
schedule  delays.  Optimized  engineering  designs 
developed  early  in  the  acquisition  process  using 


the  CREATE  tools  will  substantially  reduce  costs, 
shorten  schedules,  increase  design  and  program 
flexibility  and  agility.  Above  all,  CREATE  will  improve 
acquisition  program  performance  by  reducing 
design  flaws,  developing  sound  engineering 
designs  quickly  and  flexibly,  and  will  enable  the 
systems  integration  engineering  process  earlier 
in  the  acquisition  process. 

The  CREATE  program  will  develop  and  deploy 
three  sets  of  advanced  computational  engineering 
design  tools  for  acquisition  programs: 

1 .  military  aircraft  design, 

2.  military  ship  design,  and 

3.  antenna  design  and  integration  with 
platforms. 

These  are  illustrated  in  Figure  9. 

The  military  aircraft  design  project  will  develop 
a  design  optimization  tool  to  simulate  unsteady, 
separated  flow,  initially  for  individual  aircraft 
components  and  ultimately  for  an  entire  aircraft. 
The  military  ship  design  project  will  develop  and 
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deploy  accurate  physics-based  models  for  navy 
vessels  to  address  ship  shock  response  and 
hydrodynamics.  The  antenna  design  project  will 
build  an  efficient  electromagnetic  design  code  that 
incorporates  modern  physics  and  computational 
algorithms  for  high  performance  computers.  This 
new  generation  of  computational  design  tools  will 
enable  acquisition  system  engineers  to  rapidly 
produce  optimized  designs  for  complete  systems 
and  make  better  design  decisions  than  previously 
possible.  A  fourth  project  will  provide  software 
development  support  to  these  three  projects 
for  problem  generation,  software  engineering, 
collaboration  tools,  data  assessment  and  analysis, 
and  computational  mathematics. 

The  CREATE  projects  will  be  managed  to 
provide  enhanced  engineering  design  tools  within 
three  to  four  years  after  the  projects  begin.  Fully 
mature  tools  will  be  delivered  at  the  end  of  12-year 
project  schedule  including  two  years  for  planning 
and  project  design. 

The  CREATE  community  is  now  developing  a 
plan  for  initiating  the  CREATE  program.  That  plan 
is  based  on  the  "lessons  learned”  from  similar 
scale  projects  carried  out  by  nuclear  weapons, 
climate  modeling,  and  other  computational 
communities,  and  will  build  on  the  DoD  experience 
with  the  HPCMP  institutes  and  portfolios.  The 
CREATE  12-year  budget  is  about  $350M,  with  a 
70%  contribution  from  OSD  and  a  30%  matching 
contribution  from  the  services.  The  tri-service  T&E 
communities  will  provide  validation  experiments 
and  data  for  the  projects  as  well. 


Highlights  of  Impact  in  FY 

2006 

The  High  Performance  Computing 
Modernization  Program  provides  some  of  the 
tools  the  Department  needs  to  address  defense 
problems.  These  tools  include  modern  high 
performance  computing  hardware,  software 
and  networking.  Our  scientists  and  engineers 
use  these  tools  to  solve  many  critical  problems 
faced  by  the  Military  Departments  and  Defense 
Agencies. 

Some  problems  are  of  immediate  concern, 
while  others  are  of  longer-term  interest.  Thus, 
program  investments  impact  both  short-term  and 
long-term  issues.  The  following  vignettes  serve 
as  overviews  of  some  highlights  that  occurred  in 
FY  2006. 
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Detection  of  Moving  Targets  in 
Heterogeneous  Radar  Clutter 
Scenarios 


Surveillance  of  the  ground  by  air- 
and  space-borne  sensors  has  proven 
to  be  essential  to  the  warfighter 
and  the  intelligence  community. 
More  specifically,  the  DoD's  2006 
duadrennial  Defense  Review 
highlights  the  need  for  "a  highly 
persistent  capability  to  identity 
and  track  moving  ground  targets 
in  denied  areas''.  Ground  moving- 
target  indication  (GMTl)  radar  has 
important  advantages  over  other 
sensing  technologies  (like  optical 
sensors)  because  of  features  such 
as  day /night/all-weather  operation 
and  penetration  of  foliage, 
obscurants,  smoke  and  dust. 

However,  radar  echoes  from 
targets  have  to  compete  with 
strong  ground  clutter  returns. 
The  target  detection  performance 
of  conventional  moving  target 
indication  radar  is  strongly 
degraded  by  the  radar  platform 
motion,  due  to  the  Doppler  spread 
of  the  returns,  causing  difficulty  in 
detecting  slow  targets.  For  example, 
for  a  satellite  most  moving  targets 
are  'slow'.  Space-time  adaptive 
processing  (STAP)  is  a  signal 
and  image  processing  technique 
capable  of  compensating  for  the 
platform  motion  so  that  optimum 
detection  of  slow  moving  targets 
is  possible.  Development  and 
efficient  implementation  of  robust 
STAP  algorithms  are  particularly 
important  as  the  processing 
involves  high-dimensional 
vectors  and  matrices,  rendering 
it  computationally  intensive. 

The  core  computational  problems 
of  STAP  algorithms  usually  do  not 


have  analytical  solutions,  and  thus 
extensive  Monte  Carlo  simulations 
are  required.  Radar  systems 
operate  typically  at  false  alarm 
rates  of  ^  and  as  a  result 

performance  evaluation  using 
Monte-Carlo  methods  requires 
10^-10^^  independent  trials  for 
each  configuration  (e.g.,  different 
frequency,  azimuth  angle).  With 
the  computational  capabilities 
available  to  researchers  at  the  Air 
Force  Research  Laboratory  Sensors 
Directorate's  Flectromagnetic 
Scattering  Branch  (AFRL/SNHE), 
based  at  Hanscom  AFB,  AAA,  these 
simulations  took  days  and  even 
months  for  certain  cases  involving 
data  provided  by  the  Defense 
Advanced  Research  Projects  Agency's 
Knowledge  Aided  Sensor  Signal 
Processing  and  Expert  Reasoning 
program.  High  performance 


computing  offers  a  clear  opportunity 
for  reducing  simulation  time. 

As  part  of  a  year-long  project,  the 
PET  SIP  and  IMT  teams  helped 
to  reduce  significantly  these 
simulation  times  via  the  use  parallel 
AAATLAB®  software  and  expertise. 
For  example,  by  using  just  100 
processors  on  the  JVN  system  at  the 
Army  Research  Laboratory  MSRC, 
the  STAP  algorithms  ran  35  times 
faster.  According  to  Dr.  Freeman 
Lin  from  AFRL/SNHE: 

"On  my  PC,  the  CPU  time  for  each 
point  took  76  seconds.  This  translates 
to  the  fact  that  if  I  compnte  91  ^128 
thresholds  on  my  PC,  it  wonld  take 
almost  296  hours.  On  the  ARl  MSRC 
system,  where  you  computed  91^128 
thresholds,  it  took  7  hours,  which  is 
35  times  faster.  This  is  a  tremendous 
improvementr 


-AifplirH 


GMTl  Lynx  radar  sample  image  [from  DARPA  Affordable  Moving  Surface  Target 

Engagement] 
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Modeling  Baghdad  —  Including 
Human  Factors 


On  a  hot  summer  day  in  Iraq, 
US  soldiers  fight  a  low-intensily 
counterinsurgency  battle  on  the 
streets  of  Baghdad.  At  10  a.m.,  a 
truck  parks  near  a  warehouse  in  a 
crowded  part  of  town.  The  truck 
explodes,  killing  the  men  inside 
and  one  of  the  soldiers  standing 
guard. 

After  securing  the  area,  the 
remaining  soldiers  sound  the  alarm 
and  call  for  help.  Onlookers  gather 
—  some  cursing  the  bombers  and 
others  cursing  the  Americans  for 
attracting  the  attack.  Eventually, 
emergency  responders  arrive  and 
begin  to  treat  the  wounded  and 
quell  the  mob. 

If  it  had  occurred  in  the  real  world, 
this  scenario  generated  in  a  DoD 
simulation  would  have  immediate 
and  future  repercussions  in  the 
neighborhood,  the  country  and 
the  Middle  East. 

DoD  creates  hundreds  of  similar 
scenarios  in  the  largest  modeling 
and  simulation  environment 
that  the  Department  has  ever 
built.  The  Department  uses  the 
simulated  environment  for  a  set 
of  experiments  known  as  Urban 
Resolve  2015.  Those  experiments 
are  redefining  the  way  the  military 
operates  in  urban  environments. 
Urban  Resolve  is  also  changing 
the  way  DoD  develops  concepts, 
procures  technology  and  conducts 
training. 

The  Joint  forces  Command's 
(JECOM)  experimentation 
directorate  often  brings  new 
concepts  into  JECOM  training 


centers  to  benefit  soon-to-be- 
deployed  solders. 

Urban  Resolve  is  the  most 
important  and  complex 
experiment  conducted  since 
Millennium  Challenge  2002. 
That  2002  experiment  took  three 
years  to  plan  and  cost  about  $250 
million.  DoD  developed  Urban 
Resolve  in  half  the  time  and  spent 
about  $22  million. 

There  are  other  differences  between 
the  two.  Millennium  Challenge 
was  mostly  a  live  simulation  in 
which  14,000  people  spent  about 
three  weeks  in  the  field.  Urban 
Resolve  relies  on  virtual  operations 
in  which  people  control  computer 
entities  and  so-called  constructive 
operations  in  which  computer 
simulations  run  independently. 


But  the  biggest  differences  between 
Urban  Resolve  and  its  predecessors 
are  its  size,  architecture  and 
ambitious  intent.  With  modeling 
and  simulation  of  civilian  behavior 
still  in  its  infancy,  the  most 
difficult  aspect  of  Urban  Resolve 
is  its  model  of  human  behavior, 
which  includes  political,  military, 
economic,  social,  infrastructure 
and  information  factors. 

In  JECOM's  war  game  operations 
room,  more  than  100  DoD 
employees  and  contractors 
work  at  virtual  posts,  waiting 
for  something  to  happen.  Each 
belongs  to  one  of  three  teams:  the 
blue  team  of  coalition  forces,  the 
red  team  of  enemy  forces,  or  the 
green  team  of  Iraqi  security  forces 
and  civilians.  The  operations  room 
is  their  command  center. 


'1  mbgfi  Buiranos  £  Stn^Ames 

Multi-Resolution  Synthetic  Environment 


Aft 
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The  team  members  play  Urban 
Resolve  in  real  time  at  1 9  networked 
sites  nationwide.  Players  control 
the  blue  team  forces  from  Fort 
KnoX;  KY  and  Fort  Benning;  GA, 
Others  control  the  red  forces  from 
Fort  Belvoir,  VA;  and  the  green 
team  from  the  Space  and  Naval 
Warfare  Systems  Command  in  San 
Diego,  CA. 

Each  site  contributes  simulations 
to  Urban  Resolve.  Its  service- 
oriented  architecture  brings 
together  28  separate  simulations 
in  a  federated  command  and 
control  environment. 

Based  on  data  provided  by  the 
National  Geospatial-Intelligence 
Agency,  two  Linux-based 
supercomputers  (one  at  the 
Aeronautical  Systems  Center 
and  the  other  at  the  Maui  High 
Performance  Computing  Center) 
simulate  every  building  in 
Baghdad,  including  its  exact  size 
and  location. 


The  simulation  includes  two 
million  individual  entities,  such 
as  people  and  cars.  In  that 
environment,  people  wake  up  in 
their  homes  in  the  morning,  go 
to  work  and  create  morning  traffic 
jams.  Religious  people  pray  at 
mosques  five  times  a  day. 

Urban  Resolve  has  two  purposes. 
First,  it  will  help  DoD  develop 
new  concepts  for  fighting  in  and 
stabilizing  urban  environments.  In 
addition,  it  will  let  the  Department 
test  new  capabilities  for  winning 
conflicts  in  those  situations. 
JFCOM  chose  Baghdad  for  its 
simulation  because  it  has  a  great 
deal  of  information  about  the  cily, 
but  the  experiment  is  relevant  to 
all  urban  operations. 

Urban  Resolve  completed  three 
phases  in  2006.  In  August,  JFCOM 
pitted  a  2005  US  force— with 
current  technology  and  policies 
—against  insurgents  in  the  year 
2015.  The  experiment  identified 


planning  and  readiness  shortfalls 
and  the  risk  of  not  modernizing 
the  force.  The  experiment 
assumes  that  insurgents  in 
2015  will  have  more  dangerous 
weapons,  including  radiological, 
chemical  and  biological  agents,  in 
addition  to  better  technological 
capabilities. 

In  a  second  phase,  DoD  updated 
the  US  forces  with  capabilities 
included  in  the  Army's  budget 
plans  to  determine  whether  they 
are  sufficient.  Those  capabilities 
include  the  use  of  radio  frequency 
identification  tagging  to  track  the 
movement  of  warfighters  and 
supplies. 

In  a  third  phase,  the  US  forces 
used  experimental  concepts 
and  technologies  that  are  not  in 
DoD's  funding  plans.  Phase  3 
tested  future  concepts,  such  as 
the  Joint  Command  Post  of  the 
Future,  which  facilitates  joint 
command  and  control,  and 
the  Communications  Strategy 
Board,  which  integrates  public 
affairs  strategy  with  information 
operations  and  intelligence 
efforts. 

Urban  Resolve  represents  the 
first  time  DoD  has  used  a  full 
complement  of  military,  political, 
economic,  social  and  other 
human  factors  in  modeling  and 
simulation.  Urban  Resolve  would 
not  have  been  possible  without 
HPC. 


Urban  Setting  for  Experiments:  How  to  fight  an  asymmetric  enemy  in  2015 
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Section  2  —  Performance  Results 


FY  2006  Operations  and 

pERFORyWANCE 

The  worked  accomplished  in  FY 2006  continues 
to  assist  the  DoD  S&T  and  T&E  communities  to 
provide  support  to  the  warfighter.  Benefits  are 
both  near-term,  that  is,  within  the  present  fiscal 
year,  and  far-term. 

This  section  is  organized  by  the  goals  of  the 
HPCMP. 

Determining  the  DoD  HPCMP 
Value  to  the  Warfighter  — 
Return  on  Investment  (ROl) 

The  DoD  HPCMP  resources  are  a  fundamental 
enabling  technology  at  DoD  laboratories  and  test 
centers  for  developing  future  capabilities  and  for 
responding  to  immediate  combat  threats.  Over 
the  years,  there  have  been  many  examples  that 
confirm  that  the  HPCMP  resources  provide  high 
value  for  the  warfighter,  and  recently  this  value 
was  quantitatively  demonstrated. 

In  FY  2006,  the  HPCMP  began  an  extensive 
and  comprehensive  exploratory  process  to 
quantify  the  program  s  value  to  the  DoD  by 
conducting  a  ROl  study  of  a  subset  of  high 
importance  projects  supported  by  the  program. 
This  pilot  study  examined  the  armor/anti-armor 
lethality,  vulnerability,  and  survivability  portfolio  to 
validate  the  ROl  process  and  methodology  and  to 
determine  the  value  of  this  portfolio  to  the  DoD. 


The  Joint  Interoperability  Test  Command  was 
the  lead  in  interviewing  program  users,  acquisition 
officers,  and  theater  warfighters  to  collect 
and  analyze  data  for  this  effort.  The  National 
Defense  University  and  the  Defense  Acquisition 
University  validated  the  ROl  methodology  and 
made  recommendations  to  improve  future  ROl 
analyses. 

For  this  pilot,  the  following  projects  were 
evaluated: 

•  thermorbaric  Hellfire  [AGM-114N]  missile; 

•  Javelin  missile; 

•  electromagnetic  gun; 

•  Army  Virtual  Range  program; 

•  smart  munitions; 

•  Excalibur  guided  projectile; 

•  Modular  Artillery  Charge  System  [MACS]; 
and 

•  terminal  ballistics  applied  to  the  Armor 
Survivability  Kit  [ASK]. 

Approximately  45  people  were  interviewed, 
which  included  principal  investigators, 
computational  scientists  and  engineers,  project 
managers,  and  military  personnel  who  have  used 
technologies  in  combat  that  were  developed  with 
HPCMP  support. 

The  investment  cost  for  each  project  using 
HPCMP  centers  was  determined  by  using 
the  program  budget  allocated  for  each  major 
component  of  the  program:  HPC  Centers,  Defense 
Research  and  Engineering  Network,  and  Software 
Applications  Support  divided  by  the  number  of 
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central  processing  unit  [CPU]  hours  available  for 
that  yean  and  then  allocated  proportionally  to  each 
project  s  CPU  usage.  For  projects  using  dedicated 
systenns,  the  investnnent  was  the  funding  that 
HPCMP  provided  for  that  system. 

Categories  used  to  determine  benefits 
attributable  to  the  use  of  HPCMP  resources  are 
mission  effectiveness,  logistics,  availability,  and 
test/ experimentation  savings.  Figure  10  shows 
the  full  taxonomy  of  benefits  investigated  for  this 
analysis. 


Taxonomv  of  Benefits 

Mission  effectiveness 
Increase  lethality 
Enemy  kills 

Enemy  equipment  destruction 
Infrastructure  destruction 
Decrease  vulnerability 
Friendly  life  savings 
Friendly  equipment  savings 
Logistics  savings 
Fewer  spares 
Fewer  types  of  equipment 
Reduction  in  training  cost 
Fuel  savings 
Better  availability 

Longer  mean  time  between  failures  [MTBF] 
Shorter  mean  time  to  repair  [MTTR] 
Test/experimentation  savings 

Figure  10.  Taxonomy  of  Benefits  used  in  HPCMP  ROI 
calculations 


Hellfire  Missile 

One  project  included  in  the  ROI  analysis  that 
had  a  significant  return  on  investment  is  the 
thermobaric  Hellfire  missile.  The  Hellfire  is  a 
legacy  laser  guided  anti-armor  missile  that  was 
modified  for  urban  warfare  in  Iraq  and  Afghanistan. 
The  goal  of  this  modification  was  to  develop  a 
warhead  with  a  longer  impulse  time  than  the 
legacy  warhead  so  that  it  could  achieve  maximum 
destruction  and  lethality  within  buildings.  The 


“The  system  would  not  have  been  developed  at  all  without 
HPC  support  to  meet  required  timeline”  -  Jeff  Sinclair 
-  DTRA  Hellfire  Program;  Maj.  Gary  Harrison  -  Navy 
Hellfire  Program  Manager 


project  went  from  concept  to  deployment  in  only 
13  months,  due  to  utilizing  HPCMP  resources, 
instead  of  the  typical  30-36  months  that  would 
have  been  required  without  those  resources.  The 
system  would  not  have  been  deployed  in  time  to 
support  Operation  Iraqi  Freedom  had  HPCMP 
resources  not  been  employed  to  simulate  the 
Hellfire  warhead,  sympathetic  detonation,  and 
target  penetration.  HPCMP-enabled  simulations 
were  used  to  down  select  warhead  explosive  fill 
options  and  decrease  live  tests  planned  from  20 
to  6.  The  new  warhead  has  been  quite  effective: 
one  of  the  new  warheads  has  the  equivalent  kill 
probability  to  three  of  the  warheads  previously 
used  in  this  mission.  More  than  100  thermobaric 
Hellfires  have  been  employed  in  Iraq. 

Depending  on  the  scenario,  the  return  on 
investment  for  HPCMP-enabled  development  of 
the  thermobaric  Hellfire  is  at  least  437%  and 
could  reach  as  high  as  14,386%. 

Armor  Survivability  Kit  (ASK) 

A  second  example  of  the  HPCMP  value  to  the 
DoD  is  the  Armor  Survivability  Kit  [ASK].  This  project 
made  use  of  HPCMP-enabled  terminal  ballistics 
simulation  capabilities  that  have  been  developed 
over  the  past  ten  years.  These  simulations  have 
been  used  to  develop  and  evaluate  the  following: 
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•  novel  kinetic  energy  penetrator 
technologies; 

•  nnulti-fu notion  nnunition  technologies; 

•  novel  kinetic  energy  nnissile  lethal 
nnechanisnns; 

•  survivability  concepts  for  legacy  &  future 
connbat  systenns;  and 

•  innprovised  explosive  device  [lED]  defeat 
systenns. 

The  ASK  was  developed  to  protect  the  High 
Mobility  Multipurpose  Wheeled  Vehicle  [HMMVW] 
against  lED  threats  in  Iraq.  The  HPCMP-enabled 
ternninal  ballistics  sinnulations  allowed  rapid  down 
select  from  60  designs  to  a  "handful”  of  designs 
and  halved  the  time  required  to  deploy  the  kit  to 
the  warfighter.  Development  times  for  protection 
systems  against  evolving  lED  threats  would  have 
been  three  to  four  times  longer  without  HPCMP 
asset  availability.  In  addition,  detailed  information, 
e.g.,  first  millisecond  stress  and  deformation 
data  that  cannot  be  gathered  with  conventional 
instrumentation,  was  obtained  through 
HPCMP-enabled  simulations.  As  Lieutenant 
Colonel  [LTC]  Daniel  S.  Rusin,  US  Army,  senior 
military  engineer,  stated,  “The  first  millisecond 
of  interaction  between  bullet  and  armor  is  only 
available  using  HPC  assets  and  modeling  and 
simulation.” 

Summary 

Preliminary  summary  results  for  all  the 
projects  included  in  this  pilot  show  an  overall 
investment  of  $86  million  with  direct  benefits 
between  $487  million  and  $935  million,  yielding 
a  return  somewhere  between  463%  and  982%. 
Alternatively,  each  dollar  invested  returns  between 
$5  and  $10. 

Although  these  results  will  be  refined  as  more 
data  is  collected,  enough  information  has  been 
gathered  to  make  it  clear  that  [1  ]  HPCMP  is  more 
than  paying  its  way  in  providing  benefits  to  the 


warfighter,  and  [2]  without  HPCMP  resources, 
some  high-priority  warfighter  needs  could  not  be 
satisfied. 

Using  ROI  as  the  measurement,  the  value  of 
HPCMP  will  be  one  of  the  principal  performance 
metrics  included  in  the  annual  report  in  future 
years. 


“Without  HPC,  rapid  solutions  to  the  warfighter  within 
their  deployment  timeline  would  be  impossible”  -  LTC 
Daniel  S.  Rusin,  US  Army 
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Goal  I :  Acquire,  deploy,  operate  and  maintain  best-value 

supercomputers 


The  program  provides  high  performance 
computing  capabilities  to  the  DoD  S&T  and  T&E 
communities  through  three  modes: 

•  major  shared  resource  centers  [MSRCs]; 

•  allocated  distributed  centers  [ADCs];  and 

•  dedicated  HPC  project  investments  [DHPIsj. 

A/Vajor  Shared  Resource 
Centers  (MSRCs) 

The  MSRCs  are  very  large  centers  that  provide 
leading-edge,  high  performance  computational 
resources,  data  storage,  data  interpretation  and 
HPC  technical  expertise  to  the  defense  community. 
The  MSRCs  are  “purple”,  that  is,  they  serve  all  DoD 
Services  and  Agencies  without  regard  to  their 
location  or  supporting  organization.  They  are 
located  at  four  government  installations  are  listed 
below  and  shown  in  Figure  11 : 

•  US  Army  Research  Laboratory  [ARL], 
Aberdeen  Proving  Ground,  MD; 

•  Aeronautical  Systems  Center  [ASC],  Wright- 
Patterson  AFB,  Dayton,  OH; 

•  US  Army  Corps  of  Engineers  [USACE]  Engineer 
Research  and  Development  Center  [ERDC], 
Vicksburg,  MS;  and 

•  Naval  Oceanographic  Office  [NAVO],  Stennis 
Space  Center,  MS. 


At  the  beginning  of  FY  2006,  the  HPC  systems 
at  the  four  MSRCs  had  a  total  computational 
capability  of  113.9  teraFLOPS  [i.e.,  the  capability 
to  perform  113.9  trillion  mathematical  operations 
per  second].  During  FY  2006,  the  HPCMP 
procured  four  very  large  systems  for  deployment 
at  two  of  the  MSRCs  [ARL  and  NAVO].  These  new 
systems  have  a  computational  capability  of  109.9 
teraFLOPS.  At  the  end  of  FY  2006,  the  total 
capability  of  the  HPC  systems  at  the  four  MSRCs 
stands  at  223.9  teraFLOPS.  The  bars  in  Figure  12 
show  the  computational  growth  in  teraFLOPS  as 
well  as  Habus  atthe  four  centers  over  the  past  six 
years,  as  of  the  end  of  the  fiscal  year.  [See  callout 
on  page  11  for  a  definition  of  a  Habu.] 

Allocated  Distributed  Centers 

(ADCs) 

To  complement  the  computational  capacity 
of  the  MSRCs,  the  HPCMP  also  supports  four 
“mid-sized”  centers  that  provide  additional 
computational  resources  to  DoD  researchers. 
These  centers  are  the  ADCs.  From  the  DoD’s 
perspective,  ADCs  function  like  smaller  scale 
MSRCs  but,  in  addition  to  the  DoD,  may  serve 
other  customers  as  well.  The  four  centers  are 
listed  below  and  shown  in  Figure  11 : 

•  Arctic  Region  Supercomputing  Center  [ARSC], 

Fairbanks,  AK; 

•  Maui  High  Performance  Computing  Center 

[MHPCC],  Kihei,  HI; 
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•  Army  High  Performance  Computing  Research 
Center  [AHPCRC],  Minneapolis,  MN;  and 

•  Army  Space  and  Missile  Defense  Command 
[SMDC],  Huntsville,  AL 

In  FY  2006  the  ARSC  ADC  continued  to 
support  open  literature,  DoD  basic  research.  The 
academic  community  of  users,  whose  research 
is  supported  by  the  Offices  of  Research  in  the 
Defense  Services,  has  difficulty  quickly  obtaining 
the  access  clearances  needed  to  use  the  systems 
located  at  the  MSRCs.  This  operational  model 
allows  the  ARSC  to  mix  non-DoD  university  related 
work  and  DoD  open  literature  work  on  the  same 
systems:  a  win-win  example  of  how  the  DoD 
leverages  the  use  of  ADCs. 

Collectively,  the  ADCs  have  several  large 
HPC  systems,  including  a  952  dual-core  Opteron 
processor  SUN  system  at  ARSC  and  a  5,120  dual¬ 
core  Xeon  processor  Dell  system  at  MHPCC,  both 
added  in  FY  2006.  The  ADCs  provide  a  total  of 
91.6  teraFLOPS  [121.0  Habus]  of  computational 
capability  to  the  HPCMP.  Adding  this  computational 


power  to  the  capability  located  at  the  MSRCs,  the 
HPCMP  total  capability  increases  from  223.9 
teraFLOPS  [291.3  Habus)  to  315.5  teraFLOPS 
[412.3  Habus). 

SUMyWARY 

The  hardware  and  software  acquisition 
budget  for  the  MSRCs  over  the  last  seven  years 
has  had,  for  all  practical  purposes,  zero  growth. 
However,  in  FY  2001,  the  Program  implemented 
an  acquisition  process  whereby  all  HPCMP 
hardware  and  software  is  acquired  through 
consolidated  large  contracts  with  competitively 
selected  HPC  offerings.  The  leveraging  of  volume 
purchasing  power  combined  with  technology 
advances  commensurate  with  Moore’s  law  [a 
prediction  made  by  the  former  Chief  Executive 
Officer  of  Intel  Corporation  that  the  number  of 
transistors  contained  on  a  silicon  chip  will  double 
every  18  months)  has  provided  the  HPCMP  with 
computational  capabilities  that  exceed  traditional 
growth  curves. 


Figure  11.  Location  of  major  shared  resource  centers  [purple)  and  allocated  distributed  centers  [green) 
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Figure  12.  Growth  in  capability  of  the  MSRCs  as  of  the  end  of  the  fiscal  year 


Overarching  Governing 
Infrastructure  for  Centers 


The  HPCAAP  community  is  a  geographically  distributed  one  with  a  valuable  diversity  of  local  skills  and  capabilities  that 
must  be  captured  in  support  of  community  ends.  To  achieve  this  transfer  of  best  practices  and  innovations^  an  overarching 
centers'  governing  infrastructure  was  created: 


#  the  center  directors  of  the  four  MSRCs,  plus  ARSC,  and  MHPCC  are  members  of  the  Centers  Board  of  Directors 
(CBoD)  for  the  HPCMP  centers  capability; 


•  the  technical  specialists  that  design,  build, 
and  implement  the  solutions  comprise  the 
Engineering  Design  And  Process  Management 
Board  (ED&PMB);  and 

#  the  Distributed  Implementation  And  Operations 
Team  (DIOT),  a  group  of  individuals  at  each  of  the 
centers  are  positioned  to  sustain  the  capability. 

This  governing  infrastructure  was  developed  in  early  FY 
2004  and  chartered  in  March  of  2004.  The  adjoining 
figure  shows  the  organization  of  the  infrastructure. 
The  CBoD  has  met  several  times  to  initiate  new 
investigations  and  to  monitor  ongoing  cross-center 
intiatives.  The  other  two  teams  address  the  initiatives. 
Unified  direction  from  the  CBoD  has  helped  to  keep 
the  ED&PMB  and  the  DIOT  on  focus. 


(  DoD  HPCMP  ) 


Centers  Board  of  Directors 

Engineering  Design  and  F 

Process  Mgmt  Board 

/  ^ 

Project 

Project 
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Team  A 
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Distributed  impiementation  and  Operations  Team 


32 


Section  2  —  Performance  Results 


Goal!:  Acquire ^  develop),  dejjloy and suyyort software 
applications  and  computational  work  environments  that  enable 
critical  DoD  research,  development  and  test  challenges  to  he 

analyzed  and  solved 


The  Software  Applications  Support  [SAS] 
component  supports  Goal  2  above.  SAS  presently 
consists  of  three  major  efforts:  HPC  Portfolios, 
HPC  Software  Applications  Institutes,  and  PET, 
formerly  known  as  Programming  Environment 
and  Training.  In  FY  2007,  we  are  planning  a  fourth 
effort,  called  CREATE,  to  build  engineering  design 
analysis  tools  for  aircraft,  ships  and  antennas. 
Execution  of  CREATE  will  start  in  FY  2008.  The 
ultimate  aim  of  SAS  is  to  provide  DoD  scientists 
and  engineers  with  the  capability  for  modeling 
and  simulating  the  physical  world  to  facilitate 
the  design,  development,  test,  and  deployment 
of  superior  weapons  systems,  thereby  allowing 
our  soldiers,  sailors,  marines,  and  airmen  to  be 
prepared  better  through  training,  tactics,  and 
support  systems. 

HPC  Portfolios 

The  trend  in  research,  development,  test,  and 
evaluation  clearly  indicates  that  multidisciplinary 
problems  will  further  challenge  DoD  scientists  and 
engineers  and  require  upscale  HPC  resources. 
This  implies  that  many  of  tomorrow  s  applications 
will  incorporate  multiple  computational  disciplines, 
defined  in  this  program  by  the  CTAs.  The  portfolio 
effort  within  the  HPCMP  has  embraced  these 
needs.  Portfolios  provide  efficient,  scalable, 
portable  software  codes,  algorithms,  tools,  and 


models  and  simulations  that  run  on  a  variety  of  HPC 
platforms  and  are  needed  by  a  large  number  of 
S&T  and  T&E  scientists  and  engineers  to  execute 
their  missions.  Portfolio  development  teams  span 
DoD  Services  and  Agencies  and  include  algorithm 
developers,  applications  specialists,  computational 
scientists,  computer  scientists  and  engineers,  and 
end  users. 

Developing  software  for  scalable  HPC 
systems  remains  technically  challenging  and 
labor  intensive.  The  HPCMP  helps  the  DoD  take 
advantage  of  existing  and  future  computing  and 
communications  capabilities  by  building  software 
with  an  emphasis  on  reusability,  scalability, 
portability,  and  maintainability.  In  addition,  this 
initiative  is  producing  a  new  generation  of  world- 
class  scientists  and  engineers  trained  in  scalable 
software  techniques  to  reduce  the  future  costs 
of  doing  business  and  increase  our  defense 
capabilities.  HPC  portfolios  focus  on  specific 
themes  that  encompass  multiple  CTAs  and  cross 
Service  and  Agency  boundaries. 

The  portfolios,  illustrated  in  Figure  13  and 
described  in  the  following  paragraphs,  address 
critical  needs  in  S&T  and  T&E.  The  resultant 
software  codes  completed  in  these  efforts  provide 
DoD  scientists  and  engineers  with  applications 
software  that  efficiently  and  effectively  exploits  the 
latest  generation  of  scalable  high  performance 
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life-cycle.  The  portfolio  provides  the  warfighter 
with  fast,  state-of-the-art  analysis  by  introducing 
web-based  plug-and-play  integration  of  multi¬ 
disciplinary  codes  on  HPC  systems. 

The  portfolio’s  rotary  wing  focus  area  is 
optimizing  the  design  of  the  CH-47  and  other 
helicopters  through  higher  fidelity  predictions 
of  airloads  and  performance.  Aerodynamic  and 
structural  codes  have  been  integrated  on  HPC 
platforms,  providing  a  state-of-the-art  aeroelastic 
analysis  capability,  leading  to  aircraft  weight 
reductions. 

The  fixed  wing  focus  area  is  increasing  the 
compatibility  of  engines  with  airframes,  including 
Joint  Strike  Fighter,  through  simulations  on  HPC 


Evolution  of  Software  Applications 
Support  within  the  HPCMP 

An  evolutionary  timeline  is  shown  below  that  graphically  depicts  how  the  DoD  software  applications  programs  are  transitioning 
from  CTA  focused  activities  to  ones  that  will  lead  to  tightly  integrated,  multidisciplinary  codes  that  tackle  some  of  the  most 
comprehensive  and  complex  problems  facing  the  DoD  warfighters  today.  Both  paradigms  have  evolved  from  individual 
software  projects  for  applications  codes  from  the  mid-nineties,  where  the  efforts  focused  on  enhancing  DoD  applications  codes 
originating  up  to  several  decades  earlier.  These  codes  were  enhanced  to  become  more  robust  and  execute  efficiently  on  scalable 
hardware  coming  on  line  in  the  mid-to-late  nineties. 

From  the  beginning  of  the  software  applications  efforts 
in  1998  until  today,  the  DoD  has  completed  over  100 
projects  involving  many  hundreds  of  codes;  this  was  a 
great  boon  to  the  weapons  development,  testing,  and 
warfighting  communities.  These  efforts  improved  the 
speed,  complexity,  and  accuracy  of  military  simulations 
in  materials  for  combat  platforms,  space  and  earth 
weather  prediction,  littoral  environments,  weapons 
systems,  and  simulations  for  the  battlefield.  Codes 
released  within  the  last  few  years:  predict  the  weather 
with  forecasting  and  nowcasting;  model  radar-based 
sensing  of  surface  and  subsurface  targets,  including  land 
mines,  unexploded  ordnance,  and  vehicles;  model  3-D 
rectangular  arrangements  such  as  the  pulsed  plasma 
micro-thruster  for  microsatellite  propulsion;  model  and  simulate  large-scale  military  communications  and  tactical  signal 
intelligence  platforms,  weather  forecasting  model  improvements;  and  simulate  large  scale,  heterogeneous,  communication 
networks. 


1994 


1999 


2004 


CTA  Portfolio 

^  Serial  to  parallel  ^  Serial  to  parallel 


FOCUS \ 


►  CTA  specific 


^  Mostly  Service 
oriented  codes 


►  Technology 


^  DoD  oriented 
►  Loose  federation 


►  Some 

Cross-discipline 


^  Applications 


Now  Applications  SoUwaro  ^ 


^  Service  managed 
^  Mission  focus 


^  Parallel  enhancements  ^  High-productivity, 
high-end  computing 

^  DoD  integrated 
^  Cross-discipline 
^  Tightly  coupled 
►  Program  managed 


►  Enhanced  Service  HPC 
capability 


^  Integrated  applications 


^  Local  applications 


computing  systems.  These  applications  affect 
the  design,  acquisition,  and  utilization  of  military 
technologies  that  will  aid  in  the  development  of 
improved  military  capability  for  the  21®^  century. 

Collaborative  Simulation  and  Testing 
(CST) 

The  CST  portfolio  is  inserting  high  fidelity 
modeling  and  simulation  technologies  into 
rotary  wing,  fixed 
wing  and  airdrop 
design,  analysis, 
test  and  evaluation 
throughout 
the 

product’s 
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HPC  Software  Portfolios 


Collaborative  Simulation  and  Testing 
(CST) 

This  portfolio  provides  scalable 
software  for  military 
applications  focused  on 
collaborative  simulation  and 
testing  to  reduce  risk  in 
weapon  system 
development  and  to  provide 
information  to  senior  decision 
makers  throughout  the  life  eye 
of  the  systems.  The  products  of  the  CST 
portfolio  will  allow  for  simulations  and  resultant 
data  prior  to  a  test,  capture  test  results,  perform 
real-time  data  validation  of  test  results,  and  provide 
for  data  interrogation  and  comparisons  after  the 
test. 


Multi-Phase  Flow 
Target  Interaction 
(MFT) 

This  portfolio  delivers 
integrated, 
state-of-the-art, 
physics-based  codes  for 
the  design  and  performance 
prediction  of  munitions.  The  approach  is  a  spiral 
development,  incorporating  and  integrating  current 
understanding  of  phenomenology  into  existing, 
validated  DoD  and  DOE  component  codes  resulting 
in  an  integrated  toolkit.  The  industrial  base  will  use 
this  toolkit  to  efficiently  conceive,  design  and 
qualify  new  systems,  considering  the  full  spectrum 
of  energetics  technologies  and  applications  to  the 
United  States. 


Physics-based  Environment  for  Urban 

Operations  (PEUO)  Virtual  Electromagnetic  Design  (VED) 


This  portfolio  focuses  on  integrating  an  entity-level, 
urban  combat  model 
with  a  command  and 
control  model  and  a 
chem-bio  dispersion 
model.  The 
integrated  portfolio  is 
used  to  provide 
realistic  training 
scenarios  for  urban 
combat  operations.  The  command  and  control 
features  provide  physics-based  radio  propagation 
models  with  non-stationary  networks  while  the 
chem-bio  dispersion  models  provide  real-time, 
weather-dependent,  concentration  bands  of  toxic 
cloud  movement.  This  portfolio  will  provide 
enhanced  combat  training  as  well  as  domestic 
disaster  event,  recovery  and  relief  training. 


This  highly  integrated  portfolio  provides  the  DoD 
the  ability  to  design,  from  first-principle 
electromagnetics,  in  situ  wide-band, 

multi-functional  antennas  and 
rough  surface  scattering 
solutions  for  a  wide  range 
of  DoD  activities 
including 
communication, 
acquisition,  target 
identification,  surveillance,  and 
electronic  attack. 


Insensitive  Munitions  (IM) 

Munitions  must  reliably  fulfill  their  performance, 
readiness  and  operational  requirements  on 
demand  and  minimize  the  probability  of 
inadvertent  initiation  and  severity  of  subsequent 
collateral  damage  to  weapon  platforms,  logistic 
systems  and  personnel  when  subjected  to  unplanned  stimuli. 


Figure  13.  HPC  Portfolios 
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platforms.  High  fidelity  modeling  of  the  external 
aircraft  flow  is  integrated  with  flow  predictions 
inside  the  engine  to  provide  enhanced  performance 
through  an  improved  design  and  analysis  tool  for 
modern  fan  systems. 

Finally,  the  airdrop  focus  area  is  transforming 
the  design  of  parachute  systems  by  providing 
HPC-based  analysis  capability.  Computational 
fluid  dynamics  and  computational  structural 
dynamics  codes  have  been  integrated  to  enable 
a  low-cost  redesign  of  an  existing  parachute  to 
increase  payload  weight  with  a  minimal  increase 
in  parachute  system  weight. 

Multiphase  Flow  Target  Response 
(MFT) 

The  MFT  portfolio  is  integrating  state-of-the- 
art,  physics-based  codes  from  DoD  and  the  DOE  to 
provide  the  DoD  acquisition  community  a  common 
and  supported  toolset  that  delivers  predictions 
for  multiphase  blast  and  enhanced  blast  weapons 
effects  in  complex  military  operations  on 

urbanized  terrain 
environments. 
During  FY2006,  the 
portfolio  integrated 
multiphase  flow 
physics  into  the 
DOE  codes,  CTH, 
and  ALE3D.  A 
framework  for 
turbulent  mixing, 
gas  phase  chemistry  and  metal  combustion 
modeling  has  been  developed  and  implemented  into 
CTH  and  ALE3D.  The  portfolio  is  also  developing 
tools  for  insensitive  munitions  characterization  by 
integrating  particle  methods  and  material  models 
into  PRESTO,  a  DOE  code  designed  for  problems 
with  large  deformations,  nonlinear  material 
behavior  and  contact. 


Virtual  Electromagnetic  Design 
(VED) 

This  highly  integrated  portfolio  gives 
DoD  the  ability  to  design 
from  first-principle 
electromagnetics 
in-situ  wideband, 
multi-functional 
antennas  and  rough 
surface  scattering 
solutions  for  a 
wide  range  of  DoD 
functions  including 
communication, 
acquisition,  target  identification, 
surveillance,  and  electronic  attack.  The  VED 
portfolio  and  other  new  improved  concurrent 
electromagnetic  particle  in  cell  code  developments 
permit  end-to-end  integrated  system  modeling, 
from  pulsed  power  to  antenna  and  platform.  This 
enables  time  critical  counter-improvised  explosive 
device  simulation,  avoiding  $12M  and  18  months 
experimental  cost.  It  further  impacts  airborne 
counter  electronics  efforts,  avoiding  expensive 
flight  tests  and  reducing  electromagnetic 
compatibility/ electromagnetic  interference- 
related  problems. 

Physics-based  Environment  for 
Urban  Operations  (PEUO) 

The  PEUO  portfolio  focuses  on  integrating  an 
entity-level  urban  combat  model  with  a  command 
and  control  module  and  a  chem-bio  dispersion 
model.  The  integrated  software  will  be  used  by 
the  warfighter  training  community  to  provide 
realistic  scenarios  for  urban  combat  operations. 
The  command  and  control  features  furnish 
physics-based  radio  propagation  models  with  non¬ 
stationary  networks  while  the  chem-bio  dispersion 
models  provide  real-time,  weather-dependent, 
concentration  bands  of  toxic  cloud  movement. 
The  portfolio  has  integrated  CT-Analyst  software 
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with  the  One 
Semi-Automated 
Forces  [OneSAF] 
Objective  System 
[□□S].  CT-Analyst 
is  an  efficient 
graphical  user 
interface  for 
instantaneous 
[50ms  or  less]  plume  rendition  and  situational 
analysis.  DOS  is  a  next-generation  computer 
generated  forces  simulation.  A  demonstration 
of  the  integrated  software  has  been  provided  for 
stakeholders  from  the  US  Army  program  Executive 
Office  for  Simulation,  Training  and  Instrumentation 
who  were  quite  impressed  with  the  demonstration 
and  scenario,  which  is  running  with  a  company 
moving  through  Baghdad  and  halting  when  they 
encounter  a  smoke  cloud. 

Insensitive  Munitions  (IM) 

The  HPCMP  and  executive  agents  for  IM  within 
the  Office  of  the  Under  Secretary  of  Defense  for 
Acquisition,  Technology,  and  Logistics  sponsored  a 
workshop  to  explore  the  development  and  fielding 
of  computational  modeling  and  simulation  tools 
for  IM.  The  first  day  of  the  workshop  focused  on 
delineating  and  examining  the  state  of  existing  and 

future  IM  tools 
for  the  purpose 
of  enhancing  and 
integrating  those 
tools  to  predict 
and  minimize 
the  response 
of  munitions 
containing 

explosives  and  propellants  subject  to  unplanned 
stimuli  in  tactical  and  logistical  applications.  On 
the  second  day  of  the  workshop,  IM  technology 
area  leaders  within  DoD  formed  a  breakout 
session  to  formulate  a  strategic  direction  for  an 
integrated  HPC-based  modeling  and  simulation 


design  and  analysis  strategy  for  IM.  A  HPC 
software  development  portfolio  that  will  produce 
an  integrated  set  of  modeling  software  will  be 
selected  during  the  second  quarter  FY  2007. 

HPC  Software  Applications 
Institutes 

Institutes  address  Service/Agency  high  priority, 
high  value  technology  or  materiel  RDT&E  mission 
priorities  and  augment  traditional  processes  with 
computational  insight  by  using  legacy  or  newly- 
developed  computational  techniques.  Additional 
information  about  the  six  institutes  is  contained  in 
Figure  14  and  in  the  following  paragraphs. 

Institute  for  Maneuverability  and 
Terrain  Physics  Simulation  (IMTPS) 

The  mission  of  the  IMTPS  is  to  foster  a 
culture  within  DoD  of  using  high-fidelity  simulation 
to  attack  problems  hindering  maneuverability. 
The  IMTPS  focuses  on  simulating  near-surface 
environmental  processes  to  support:  1  ]  detection 
of  landmines,  improvised  explosive  devices,  and 
unexploded  ordnance;  2]  use  of  seismic  and 
acoustic  unattended  ground  sensor  networks; 

3]  analysis  of  maneuverability  and  trafficability;  and 

4]  remote  sensing  of  denied  areas.  The  IMTPS 
vision  integrates  physics-based  geotechnical, 
geophysical,  hydrogeologic,  and  hydrologic 
analyses  into  a  virtual-testing  facility  for  resolving 
terrain-related  warfighter  problems. 

During  FY  2006,  the  IMTPS  developed 
conceptual  modeling  of  an  urban  environment 
that  allows  the  definition  of  buildings,  terrain, 
sub-surface  soil  layers,  and  air  layers,  which  can 
be  processed  to  provide  information  for  a  finite 
difference  acoustic  analysis.  Buildings  may  be 
defined  from  two-dimensional  [2-D]  extrusions  of 
floor  footprints  defined  by  a  geographic  information 
system  or  by  computer  aided  drafting  and  design 
data.  The  soil  and  air  layering  may  be  defined  using 
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triangulated  irregular 
networks  or 
digital  elevation 
nnodels.  The 
urban  nnodeler  allows 
easy  alignnnent  of  buildings 
with  terrain  features  and 
buildings  may  be  placed  on 
arbitrary  terrain  surfaces. 
Material  properties  can  be 
specified  interactively  from  a 
user  defined  material  property 
table.  Underground  facilities  and 
tunnels  can  also  be  modeled  as  extrusion  of  2-D 


footprints.  The  modeler  is  capable  of  processing 
a  virtually  unlimited  number  of  buildings  and 
soil  layers  to  provide  finite  difference  meshes 
consisting  of  hundreds  of  millions  of  nodes.  A 
major  enhancement  to  current  capabilities  is 
the  ability  to  model  complicated  building  shapes 
that  can  be  non-orthogonal  with  concavities,  thus 
allowing  real  world  cities  to  be  analyzed.  The 
urban  modeler  provides  enhanced  modeling 
capabilities  compared  to  current  methods  of 
model  generation  and  reduces  the  time  required 
for  model  generation.  This  model  will  be  used 
to  support  Army  evaluation  of  acoustic  sensors 
operating  in  an  urban  environment. 


Biotechnology  High-Performance 
Computing  Software  Applications 
Institute  for  Force  Health  Protection 
(BHSAl) 

The  BHSAl  serves  as  an  interdisciplinary,  tri- 
Service  resource  to  develop  and  apply  HPC  software 
that  will  accelerate  research  and  development  of 
militarily  necessary  medical  products  for  DoD  s 
Force  Health  Protection  strategy.  In  force 
protection,  the  ability  to  rapidly  differentiate 
between  benign  and  pathogenic  organisms  is  vital 
for  battlefield  treatment.  The  differences  between 
a  benign  and  pathogenic  organism  are  impossible 
to  distinguish  by  eye,  necessitating  chemical  and 


require  the  same  intense 
medical  intervention.  During 
FY  2006,  the  institute 
completed  development  of 
tools  to  examine  sequence 
fragments  and  identify 
short,  specific  and  unique 
sequences  to  an  organism, 
termed  “fingerprints”.  The 
novel  strategy  employed  by 
the  BHSAl  is  to  exploit  small 
differences  that  do  not  exist 
in  similar  organisms.  The  US 
Army  Medical  Research  Institute  of  Infectious 
Diseases  has  used  the  BHSAIs  diagnostic  assays 
to  create  printed  custom  arrays  incorporating  in- 
silico-determined  fingerprints  able  to  differentiate 
Y pestis  from  Y.  pseudotuberculosis. 


biological  diagnostic  assays.  The  problem  to 
be  solved  is  how  to  distinguish  between  closely 
related  species  where  one.  Yersinia  pestis  causes 
plague,  but  another  Yersinia  pseudotuberculosis 
is  less  harmful  and  does  not 


Battlespace  Environments  Institute 
(BED 


The  BEI  migrates  existing  DoD  existing 
climate/weather/ocean  modeling  and  simulation, 
environmental  quality  modeling  and  simulation, 
and  space  weather  applications  to  the  Earth 
System  Modeling  Framework  [ESMF]  and  assists 
in  transitioning  non-DoD  ESMF  applications  to  DoD. 
During  FY  2006,  BEI  used  the  ESMF  to  couple  the 
Navy  Coastal  Ocean  Model  and  Coupled  Ocean/ 
Atmosphere 
Mesoscale 
Prediction 
System  as 
a  single 
executable 
application. 

The  flexible 
software 
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HPC  Software  Applications 

n  Institutes  IMTPS 


ISSA 


1 


lach  institute  has  a 
critical  mass  of  experts 
keeniy  focused  on  using 
computationai  science 
and  high  performance 
computing  to  accelerate 
solving  the  Departmenfs 
highest  priority 
challenges.  With 
cross-Service  and  Agency 
teaming  and 
multi-disciplinary 
approaches,  the  institutes 
transform  traditional 
operational  processes 
with  computational  insight 
by  using  legacy  or 
newly-developed 
computational  tools. 


HI-ARMS 


IHAAA 


4  ' 

11- 


Institute  for  Maneuverability  and  Terrain  Physics  Simulation  (IMTPS) 

The  institute  focuses  on  simuiating  near-surface  environmentai  processes  to 
support:  detection  of  iandmines,  improvised  expiosive  devices,  and  unexpioded 
ordnance;  the  use  of  unattended  ground  sensor  networks;  anaiysis  of  maneuver  and 
traffic-abiiity;  and  remote  sensing  of  denied  areas. 

Biotechnology  HPC  Software  Applications  Institute  for  Force  Health 
Protection  (BHSAI) 

The  institute  buiids  HPC  experience  and  expertise  within  the  DoD  to  deiiver  the  best 
medicai  and  non-medicai  biotechnoiogy  soiutions  to  protect  and  treat  our 
warfighters. 

Battlespace  Environments  Institute  (BEI) 

This  institute  migrates  existing  DoD  ciimate/weather/ocean  modeiing  and  simuiation, 
environmentai  quaiity  modeiing  and  simuiation,  and  space  weather  appiications  to 
the  Earth  System  Modeiing  Framework  (ESMF)  and  assists  in  transitioning  non-DoD 
ESMF  appiications  to  DoD. 

HPC  Software  Applications  Institute  for  Space  Situational  Awareness  (ISSA) 

The  institute  addresses  four  top  priority  capabiiity  shortfaiis  in  the  Space  Situationai 
Awareness  community:  astrodynamics,  image  enhancement,  non-imaging  space 
object  identification,  and  knowiedge  fusion.  The  institute  appiies  the  power  of  HPC 
and  advanced  aigorithms  to  identify  the  functionaiity,  capabiiity,  mission,  status,  and 
heaith  of  space  objects. 

Institute  for  HPC  Applications  to  Air  Armament  (IHAAA) 

This  institute  identifies  and  integrates  new  technoiogies  and  rebuiids  and 
restructures  existing  Service-generated  software  using  formai  software  engineering 
procedures  that  wiii  buiid  acquisition  community  confidence.  Greater  accuracy  and 
rapid  production  of  HPC  soiutions  wiii  enabie  eariy  detection  of  probiem  areas  in 
new  systems  and  provide  quicker  reaction  to  warfighter  needs. 

HPC  Institute  for  Advanced  Rotorcraft  Modeling  and  Simulation  (HI-ARMS) 
This  institute  significantiy  increases  domestic  capabiiity  to  anaiyze  and  design  future 
rotorcraft  systems  to  meet  heavy-iift  requirements  of  the  Department  of  Defense. 
Institute  software  products  are  buiit  according  to  the  physicai  accuracy,  soiution 
throughput  and  cost,  and  soiution  quaiity  priorities  necessary  to  create  a  rotorcraft 
design  process  around  HPC. 


Figure  14.  HPC  Software  Applications  Institutes 
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design  provides  a  basis  for  advanced  developnnent 
of  two-way  coupling,  which  will  be  connpleted  by 
the  end  of  FY  2007  The  two-way  coupled  ocean- 
atnnosphere  prediction  systenn  will  provide  realistic 
feedback  at  the  air-sea  interface;  leading  to  nnore 
accurate  predictions  for  the  warfighter  in  littoral 
and  deep-water  areas.  It  nnay  also  be  used  for 
innproved  hurricane  prediction,  ocean  prediction, 
optimunn  track  ship  routing,  search  and  rescue, 
anti-subnnarine  warfare  and  tactical  planning. 


Institute  for  HPC  Applications  to  Air 
Armament  (IHAAA) 


The  IHAAA  integrates  HPC  with  the  need  to 
field  new  weapons  and  new  weapon  configurations 
in  a  rapidly  changing  warfare  environnnent. 

Through  its  various  projects,  the 
IHAAA  delivers  faster  analysis 
capability  that  translates  to  faster 
developnnent  and  certification  of  air 
arnnannent  systenns.  Sonne  of  the 
acconnplishnnents  during  FY  2006 
include:  the  successful  application 
of  HPC  tools  to  help  solve  a 
tailboonn  nnodification  of  the  UH- 
60  to  correct  a  hover  problenn 
in  high  cross  wind  conditions 
that  avoided  $200,000  in 
flight  test  costs;  the  tinnely 
support  of  a  B-52  safety 
investigation  board  using  HPC  tools,  collaboration 
between  Naval  Air  Systenns  Connnnand,  Northrop- 
Grunnnnan,  and  the  Air  Force  Seek  Eagle  Office 
that  resulted  a  $1.17M  cost  avoidance  for  pod 
integration  on  F-18;  expansion  of  HPC  application 
to  aircraft-store  connpatibility  analysis  in  stability 
and  control  and  flutter;  and  application  of  HPC 
capability  to  integrate  the  new  Miniature  Air 
Launched  Decoy  on  the  B-52.  Many  of  these 
acconnplishnnents  reduce  delivery  tinne  for  new 
war  fighting  capability,  reduce  costs  to  deliver 
this  capability,  or  reduce  linnitations  to  systenn 
ennploynnent. 


Institute  for  Space  Situational 
Awareness  (ISSA) 

The  ISSA  supports  the  space  situational 
awareness  needs  of  warfighters  by  developing  HPC 
software 
applications. 

SSA  includes 
the  space 
support 
and  nnission 
support 
foundation 
tiers  of  the 
United  States  military  space  power,  and  directly 
supports  the  missions  of  offensive  and  defensive 
counter-space.  The  ISSA  is  increasing  combat 
capability  by  providing  sharper,  high  resolution 
optical  imagery  of  space  objects  of  interest  for 
space  control  and  SSA.  During  FY  2006,  the  ISSA 
developed  SIMFENCE,  a  modeling  and  simulation 
tool  that  simulates  Space  Surveillance  Network 
[SSN]  sensors  and  aids  space  surveillance 
systems  development,  architecture,  and 
operational  decision-making  processes.  The  ISSA 
has  also  provided  enhanced  capability  to  analyze 
and  architect  SSN  force  structure,  considering 
optical,  infrared,  and  radar  sensors;  satellite 
catalog  accuracy;  loss  satellite  rates;  and  event 
detection  timeliness  by  using  HPO  to  improve 
simulation  times  on  the  SSN  Analysis  Model.  The 
improvements  will  provide  rapid  SSN  analysis  to 
the  warfighter.  In  the  area  of  image  enhancement, 
the  ISSA  has  developed  Physically-Constrained 
Iterative  Deconvolution  [PCID]  software,  which 
provides  images  that  approach  the  theoretical 
limit  of  image  quality.  Dramatic  decreases  in 
execution  times  have  been  realized  through 
software  engineering  and  employment  of  HPC, 
thus  making  PCID  viable  for  users. 
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HPC  Institute  for  Advanced 
Rotorcraft  Modeling  and  Simulation 
(Hl-ARMS) 

The  mission  of  Hl-ARMS  is  to  transform 
the  analysis-test  paradigm  that  currently  exists 
within  the  rotorcraft  industry  and  government 

laboratories 
in  the  United 
States  into 
one  built 
around 
HPC,  which 
will  provide 
domestic 


manufacturers  the  means  to  create  effective 
designs  [or  upgrades]  of  rotorcraft  systems 
required  by  DoD  and  to  minimize  development 
cost  and  risk.  Simultaneously,  the  DoD  will  have 
the  means  to  accurately  predict  mission  capability, 
to  improve  the  effectiveness  of  vehicle  test 
programs,  and  to  effectively  conduct  rotorcraft 
source  selection  processes,  including  analyses 
required  to  support  airworthiness  qualification. 
Currently,  HI-ARMS’s  work  is  supporting 
investigation  of  helicopter  brownout  which  occurs 
when  operating  in  unimproved  landing  areas,  such 
as  Middle  East  war  zones.  Improved  prediction 
and  understanding  of  brownout  flow  fields  can 
reduce  the  possibility  of  aircraft  incidents  and  lead 
to  mitigation  through  aerodynamic  design. 


Physically  Constrained  Iterative 
Deconvolution  [PCID]  Algorithm 


Researchers,  sponsored  by  the  Air  Force  Office  of  Scientific  Research,  invented  the  Physically  Constrained  Iterative  Deconvolution 
(PCID)  algorithm  and  proved  that  this  algorithm  achieves  the  theoretical  limits  to  image  quality  as  calculated  using  the 
Cramer-Rao  lower  bound  of  Fisher  information  theory. 


PCID  is  an  iterative  image  resortation  algorithm.  It  estimates  and  removes 
atmospheric  and  system  blurring  from  one  or  more  frames  of  blurred  and  noisy 
measured  data  to  produce  a  single  high-resolution  image.  It  achieves,  or  when 
less  computational  effort  is  desired,  closely  approaches  the  theoretical  limits  to 
image  quality.  That  is,  PCID  extracts  the  most  amount  of  information  that  can 
be  extracted  from  image  data,  and  in  this  sense  no  better  algorithm  can  ever  be 
invented] 

The  research  surrounding  PCID  ended  the  '"super-resolution"  controversy  which 
had  been  raging  for  nearly  a  half  century.  It  remains  an  open  basic  research 
question  whether  it  is  possible  to  create  an  algorithm,  meeting  the  Cramer-Rao 
lower  bound,  which  has  fewer  mathematical  operations,  and  thus  might  be  more 
computationally  efficient. 

Armed  with  this  essential  fundamental  knowledge,  the  DoD  determined 
sufficient  computational  power  existed  to  warrant  investment  in  a  large  software 
development  effort  to  produce  an  engineering  tool  which  would  be  fast  enough 
to  provide  timely  operational  SSA  to  the  warfighting  commands.  Thus,  the  ISSA 
was  founded  and  funded. 


No  processing 


The  figure  on  the  right  is  an  example  of  the 
defraction  limited  image.  The  figure  on  the 
left  shows  the  results  of  processing  with 
PCID.  This  data  consists  of  300  images 
taken  with  a  1.6m  telescope,  and  using  four 
hours  computation  time  on  a  64  processor 
HPC  system. 
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User  Productivity 
Enhancement  and  Technology 
Transeer  (PET) 

PET  enables  the  DoD  HPC  userconnnnunityto 
nnake  the  best  use  of  the  connputing  capacity  the 
progrann  provides  and  to  extend  the  range  of  DoD 
technical  problenns  solved  on  HPC  systenns.  PET 
is  enhancing  the  total  capability  and  productivity 
of  the  progrann  s  user  connnnunity  through  HPC- 
related  science  and  technology  support,  training, 
collaboration,  tool  developnnent,  support  for 
software  developnnent,  technology  tracking, 
technology  transfer,  and  outreach  to  users. 

PET  is  responsible  for  gathering  and  deploying 
the  best  ideas,  algorithms,  and  software  tools 
emerging  from  the  national  HPC  infrastructure 
into  the  DoD  user  community.  The  PET  activities 
are  conducted  through  two  separate  contracts; 
one  to  MOS  University  Consortium,  led  by 
Mississippi  State  University,  and  the  second  to 
High  Performance  Technologies,  Incorporated. 
The  teams  from  both  contracts  involve  academic 
leaders  to  serve  as  points  of  contact  for  each 
of  the  areas  covered  by  PET  and  experienced 
Ph.D.-level  personnel  located  at  DoD  sites  to 
provide  HPC  and  one-to-one  scientific  assistance 
to  HPCMP  users.  The  teams  are  comprised  of 
experts  from  a  broad  range  of  universities  and 
companies  highly  regarded  in  the  HPC  field  [see 
Figure  15].  In  addition,  PET  personnel  lead  short¬ 
term  projects  that  focus  on  delivering  capabilities 
for  specific  needs. 

PET  supports  all  ten  HPCMP  computational 
technology  areas,  and  the  following  four 
crosscutting  areas,  with  a  broad  HPCMP-wide 
management  approach. 


Enabling  Technologies  (ET) 

The  ET  functional  area  provides  tools, 
algorithms,  and  standards  for  pre-  and 
post-processing  large  datasets.  Such  processing 
includes  the  following  technologies:  mesh 
generation,  visualization  [both  local  and  remote], 
data  mining  and  knowledge  discovery,  image 
analysis,  and  problem  solving  environments. 

Computational  Environment  (CE) 

Improving  the  usability  of  the  computational 
environments  at  the  HPCMP  centers  is  critical  for 
easily  and  effectively  using  the  program  s  resources. 
CE  includes  all  aspects  of  the  users  interface  to 
high  performance  computing  resources,  such  as 
programming  environments  [debuggers,  libraries, 
solvers,  higher  order  languages;  performance 
analysis,  prediction,  and  optimization  tools], 
computing  platforms  [common  queuing,  clusters, 
distributed  data,  and  metacomputing],  parallel 
algorithms,  user  access  tools  [portals  and  web- 
based  access  to  high  performance  computing 
resources],  and  consistency  across  the  centers 
for  locating  these  capabilities. 

Collaborative  and  Distance  Learning 
Technologies  (CDET) 

This  functional  area  focuses  on  supporting 
HPCMP  users  who  are  unable  to  attend  HPC- 
based  events,  such  as  training  classes  and 
meetings.  CDLT  is  responsible  for  webcasting 
and  video-capturing  events  and  post-processing 
the  material  to  create  high  quality  instructional 
content.  After  approval,  such  content  is  available 
for  downloading  from  the  PET  Online  Knowledge 
Center.  CDLT  also  provides  support  for  video 
teleconferencing  services.  Strong  interactions 
with  the  OPEN  component  and  with  Centers’  staffs 
ensure  that  CDLT  activities  are  coordinated  and 
incorporated  into  the  program’s  networking  and 
security  infrastructure. 
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Education,  Outreach,  and  Training 
Coordination  (EOTC) 

This  functional  area  coordinates  fornnal 
and  infornnal  knowledge  delivery  to  the  DoD 
HPCMP  user  connnnunity  and  outreach  to  other 
connnnunities.  EOTC  enconnpasses  PET-sponsored 
HPC-based  training,  sunnnner  intern  progranns, 
sunnnner  institutes  at  nninority  serving  institutions 
[MSI],  visiting  faculty  progranns,  and  general  HPC 
outreach.  EOTC  provides  opportunities  for  MSI 
staff,  faculty  and  students;  undergraduate  and 
graduate  students:  postdoctoral  and  visiting 
faculty  appointnnents:  and  the  training  of  future 
DoD  HPCMP  users.  Work  in  this  functional  area 
includes:  coordinating  on-site  training  at  the 
progrann  s  shared  resource  centers  and  rennote 
sites:  selecting  optinnal  training  delivery  nnethods 
and  nnedia;  coordinating  outreach  forunns,  such 


as  conferences,  workshops,  senninars,  and 
synnposia:  establishing  and  nnaintaining  a  coherent 
framework  to  integrate  undergraduate,  graduate 
students,  postdoctoral  and  visiting  faculty  into 
the  PET  activity;  and  developing  programs  and 
activities  that  promote  careers  in  computational 
science  and  high  performance  computing. 

PET  Highlights 

HPCMP  technical  and  program  management 
has  emphasized  and  encouraged  our  entire  team 
of  functional  experts,  on-site  personnel,  principal 
investigators,  and  business  administrators  to 
focus  on  the  key  goals  of  PET  program:  technology 
transfer,  user  productivity,  and  DoD  mission 
impact.  The  following  example  shows  such  an 
achievement. 
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Scaling  Security  Code  for  Execution  on  Larger 

Linux  Clusters 


Problem 

Data  obfuscation  is  emerging  as  a  key  technology  for  enabling  battle-space  communication  dominance.  This 
technology  allows  friendly  forces  to  send  information  securely  without  it  being  detected,  decoded,  or  modified. 
It  also  provides  friendly  forces  with  the  ability  to  detect,  decode,  and  modify  enemy  communications.  Air  Force 
Research  Laboratory,  Information  Directorate  (AFRL/IF)  researchers  Chad  Heitzenrater  and  Zenon  Pryk  have 
been  conducting  research  in  the  area  of  data  obfuscation  with  the  goal  of  enhancing  the  performance  of  these 
DoD  capabilities  through  the  use  of  high  performance  computing.  While  the  local  Rome,  NY,  research  site's 
ten-processor  Linux  cluster  is  invaluable  for  development  purposes,  it  is  limited  in  its  ability  to  assist  in  solving 
larger  and  more  realistic  problems  of  DoD  interest.  A  request  to  the  HPCAAP  resulted  in  a  large  allocation  on 
one  of  the  HPCMP's  premiere  classified  systems,  Stryker,  located  at  ARL.  Unfortunately,  neither  Heitzenrater 
nor  Pryk  had  access  to  the  system  because  there  is  no  available  classified  DREN  connection  at  AFRL/IF.  As  a 
result,  the  AFRL  researchers  found  themselves  in  a  challenging  situation  with  regard  to  continuing  their  HPC 
research  activity. 

Methodology 

PET  Team  members  Bill  Yurcik  and  Paul  Sotirelis  met  with  the  AFRL/IF 
researchers  in  August  2006  and  developed  a  plan  to  assist  Heitzenrater 
and  Pryk  in  using  Stryker,  Yurcik  arranged  a  meeting  at  ARL  where 
Heitzenrater  provided  a  brief  introduction  to  his  work  and  coordinated 
support  from  another  member  of  the  PET  contracting  team,  Jim  Eischer, 

Eischer  was  able  to  build  the  data  obfuscation  code  and  begin  running 
jobs  on  Stryker.  Throughout  November  and  December  2006,  an  execution 
environment  capable  of  running  a  comprehensive  set  of  test  cases  was 
developed  where  nearly  64,000  CPU  hours  were  utilized. 

Users  Supported 

In  addition  to  Heitzenrater  and  Pryk,  other  DoD  personnel  performing  classified  security-related  processing  may 
benefit  from  the  capability  to  scale  their  code  on  larger  Einux  clusters, 

DoD  Impact 

Earger  and  more  DoD  relevant  data  obfuscation  problems  may  now  be  explored  due  to  PET's  support  in  enabling 
AERUs  access  to  its  HPCAAP  allocation.  The  project  in  total  has  over  400,000  hours  allocated  to  it  and  it  is 
expected  that  now  they  will  be  able  to  utilize  these  hours  as  intended.  Heitzenrater  wrote,  "The  work  performed 
by  the  PET  group  has  helped  to  fill  a  critical  gap  in  a  crucial  technology  area.  Prior  to  their  help,  I  was  able  only  to 
test  on  local  clusters,  [and  was]  limited  in  scope  to  a  maximum  of  20  nodes.  Preliminary  data  generated  by  the  team 
has  been  analyzed,  and  has  provided  valuable  input,  which  is  being  rolled  into  the  final  technical  report  for  the  High 
Performance  Implementation  of  Data  Obfuscation  Technology  program.  Without  this  help  I  would  not  have  been  able 
to  test  this  project  to  the  scale  it  was  designed  to  achieve,  and  without  any  other  resources  available,  I  would  not  be 
able  to  say  nearly  as  much  about  the  capability  this  program  has  achieved," 
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Goal  3;  Acquire,  deploy,  operate  and  maintain  a  communications 
network  that  enables  effective  access  to  supercomputers  and  to 
distributed  SSrT/TSrE  computing  environments 


Defense  Research  & 
Engineering  Network  (DREN) 

DREN  was  created  to  link  high  perfornnance 
connputing  users  and  superconnputers,  no  nnatter 
where  the  person  or  resource  is  or  with  what 
Military  Service  they  are  associated  [see  Figure 
16].  Fronn  the  beginning,  DREN  has  acted  as 
an  enabler  for  the  research,  developnnent,  test 
and  evaluation  connnnunities,  the  Missile  Defense 
Agency  [MDA],  DoD  Modeling  &  Simulation  Office, 
Joint  Forces  Command  [JFCOM],  Defense  Threat 
Reduction  Agency,  and  others. 

DREN  enables  MSRC  to  perform  secure, 
large-scale,  remote,  mass-storage  for  HPC 
disaster  recovery.  Although  it  s  always  been  highly 
desirable  to  do  in-band  [live  on-line]  mass  storage 
transfers,  it  was  in  the  “too  hard  to  do”  category. 
The  challenge  of  transferring  terabits  of  data 
daily  between  multiple  centers  was  out  of  reach. 
Recently,  a  number  of  advances  have  made  these 
types  of  data  exchanges  a  reality.  Access  to  the 
DREN  backbone  was  expanded  at  each  of  the  DoD 
major  shared  resource  centers  to  optical  carrier 
or  OC-48  [approximately  2.4  gigabits  per  second]. 
These  centers  are  the  first  within  the  DoD  to 
have  massive  wide-area  network  [WAN]  access 
capabilities.  Anticipating  rapidly  rising  bandwidth 
demands,  DREN  revamped  its  backbone  nationwide 
using  new  protocol  architectures  [multi-protocol 
label  switching  and  internet  protocol  security 
tunnels]  with  jumbo  frame  enabled  internet 


protocol  which,  in  turn,  permits  high-end  tuning 
of  computational  resources  over  thousands  of 
miles  for  massive  data  transfers. 

DREN  is  centrally  funded  for  science  and 
engineering  users  of  DoD  high  performance 
computational  resources.  Other  congressionally 
authorized  groups  [MDA,  modeling  &  simulation, 
operational  test  and  evaluation  groups]  not  part  of 
HPC  line-item  funding  must  offset  service  delivery 
point  and  security  costs  to  access  the  DREN. 

It  is  in  the  best  interest  of  the  DoD  to 
continuously  expand  the  pool  of  quality  scientists 
and  engineers  working  on  high  priority  DoD 
problems.  Potential  new  users  often  discover 
the  availability  of  HPC  resources  through 
initial  exposure  to  DREN.  JFCOM  in  Suffolk,  VA 
followed  this  pattern  and  eventually  expanded 
into  a  joint,  distributed,  system-of-systems  virtual 
communications  concept  for  future  real-time 
communications  and  network  simulations.  An 
advantage  of  DREN  is  that  it  makes  high  capacity 
bandwidth  available  to  all  computational  resources 
wherever  they  may  be.  This  approach  makes 
it  much  easier  to  ensure  optimal  use  of  high 
performance  computing  assets  and  reduces  the 
effective  cost  of  these  scarce  resources. 

Historically,  we  associated  access  to  scarce 
and  expensive  resources  with  close  proximity 
to  major  centers  of  civilization.  Today,  we  have 
much  more  flexibility  in  the  placement  of  new 
computational  resources.  That  flexibility  allows 
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Figure  16.  DREN  connections,  HPC  centers  and  other  network  access  points 
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growth  of  new  skill  and  job  opportunities  to  rural 
[Gulf  Coast,  Midwest,  and  Southwest]  or  rennote 
[Alaska  and  Hawaii]  labor  nnarkets  that  otherwise 
would  be  overlooked.  High  bandwidth  WAN  access 


allows  the  HPCMP  to  get  resources  very  close  to 
specialized  real-tinne  systenns  while  expanding  the 
pool  of  potential  users  working  on  DoD  problenns 
and  keeps  those  resources  extremely  busy. 


Goal  4;  Fromote  collaborative  relationships  among  the  DoD 
computational  science  community,  the  national  computational 
science  community  and  minority  serving  institutions  (AASls) 


Defense  Research  & 
Engineering  Network  (DREN) 

As  one  of  the  three  major  areas  of  DoD’s 
high  performance  computing  modernization 
program,  DREN  draws  from  the  high  performance 
computing  community  most  familiar  with  Defense 
supercomputing  for  technical  and  security  advisory 
group  members.  DREN  personnel  also  participate 
in  the  more  generalized  DoD  networking  and 
security  communities  within  the  Global  Information 
Grid,  through  direct  participation  on  DoD  control 
boards  and  technical  advisory  councils,  and  by 
participating  as  a  Tier  2  DoD  Computer  Emergency 
Response  Team  for  hostile  acts  of  intrusion  and 
compromise. 

DREN  contributes  to  overall  federal  agency 
networking  and  security  through  the  Large  Scale 
Network  [LSN]  and  Joint  Engineering  Team 
[JET].  These  groups  maintain  and  extend  US 
technological  leadership  in  leading-edge  network 
technologies  and  coordinate  federal  agency 
networking  activities,  operations,  and  plans 
represented  by  DoD  DREN,  Department  of  Energy, 


National  Aeronautics  and  Space  Administration, 
the  National  Science  Foundation,  Next  Generation 
Internet,  and  Internet  2.  The  JET  and  LSN  are 
part  of  the  White  House  s  Office  of  Science  and 
Technology  Policy  Interagency  Working  Group. 

DREN  peers  [exchanges  network  traffic]  at 
international  exchange  points  including  Starlight 
in  Chicago,  and  the  Pacific  Northwest  Gigapop 
in  Seattle,  and  at  advanced  exchanges  including 
Next  Generation  Internet  Exchanges  East  and 
West  in  Maryland  and  California.  DREN  actively 
participates  in  international  science  exchanges 
such  as  the  Australian  Meteorological  and 
Oceanographic  Society  and  Asian  Pacific  Advanced 
Networks  projects. 

User  Productivity 
Enhancement  and  Technology 
Transfer  (PET) 

The  EOTC  functional  area  within  PET  is 
responsible  for  creating  education  opportunities 
targeted  to  undergraduate  and  graduate 
education,  with  emphases  on  MSIs,  by  sponsoring 
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summer  intern  programs  and  summer  institutes. 
A  goal  is  to  create  a  workforce  pipeline  for  the 
Department  of  Defense  and  the  nation. 

The  Summer  Intern  program  takes  place 
in  June,  July,  and  August.  The  Summer  2006 
Intern  program  was  successful,  and  the  student 
presentations  are  in  the  EOTC  section  on  the 
PET  Online  Knowledge  Center  [OKC]  [https://okc. 
erdc.hpc.mil].  From  these  presentations  we  get 
the  clear  message  that  not  only  do  the  students 
gain  valuable  experience  in  a  DoD  laboratory 
environment,  but  the  projects  on  which  they 
work  directly  impact  DoD  research.  A  total  of  34 
summer  interns  from  26  universities  were  placed 
at  six  locations:  ARL-Aberdeen,  MD  [7  interns]; 
ERDC-Vicksburg,  MS  (8  interns];  ASC-Wright- 
Patterson  AFB,  OH  [14  interns];  NRL-Stennis 
Space  Center,  MS  [2  interns];  NRL-Washington, 
DC  [2  interns];  and  Air  Force  Weather  Agency, 
Offutt  AFB,  NE  [1  intern]. 

One  of  our  primary  efforts  in  attracting 
and  preparing  students  at  MSIs  for  the  intern 
program  is  the  summer  institute  program. 
The  summer  institute  program  is  comprised 
of  a  two-week  event  at  each  of  the  four  MSIs. 
Each  institute  introduces  students  to  HPC  and 
provides  introductory  instruction.  In  the  Summer 
of  2006,  PET  sponsored  institutes  at  Jackson 
State  University,  Florida  International  University, 
University  of  Hawaii,  and  Central  State  University. 
Fifty  students  attended  the  summer  institutes 
where  they  collaborated  with  PET  personnel  from 
several  functional  areas,  thus  giving  them  a  well- 
rounded  experience,  including  the  opportunity  to 
present  their  work  [see  Figure  17]. 

The  Computational  Science  Workshop  for 
Underrepresented  Groups  was  held  again  in 
January  2006  on  the  campus  of  the  University 
of  Southern  California  [see  Figure  18].  This 
annual  event,  jointly  supported  by  PET  and  other 
organizations,  brings  together  students  and  faculty 


Figure  17.  Group  of  students  that  participated  in  the 
summer  institute  at  Central  State  University 


from  MSIs  for  a  week-long  course  on  building  a 
parallel  computer,  and  on  methods  for  solving 
problems  in  computational  science. 

In  2006,  the  following  MSIs  participated  in 
PET  education  and  technology  transfer  activities: 
Alabama  A&M  University,  Central  State  University, 
Florida  International  University,  Howard  University, 
Jackson  State  University,  University  of  Hawaii, 
North  Carolina  A&T  University,  University  of 
Southern  California,  University  of  Texas  at  El  Paso, 
and  University  of  Texas  at  San  Antonio. 


Figure  18.  Participants  at  the  2006  Computational  Science 
Workshop  for  Underrepresented  Groups 
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Goal  5:  Continuously  educate  the  RDTSrE  workforce  with  the 
knowledge  needed  to  employ  computational  modeling  effectively 

and  efficiently 


User  Productivity 
Enhancement  and  Technology 
Transeer  (pet) 

The  PET  contracts  offered  55  training  events 
this  past  yean  attended  by  802  students,  covering 
subjects  ranging  fronn  code  profiling  and  error 
estimators  to  user  training  on  codes  such  as 
FLUENT  ABAQUS®,  EnSight,  and  Xpatch®.  See 
Table  4  for  a  sampling  of  courses  given  in  FY 
2006. 


Many  PET  courses  are  captured  on  video  and 
can  be  downloaded  from  the  PET  OKC  [https:// 
okc.erdc.hpc.mil  ]  onto  the  users’  desktops  and 
viewed  at  their  leisure. 

While  the  OKC  contains  online  training 
opportunities,  it  also  is  a  repository  of  PET- 
developed  technical  reports,  presentations, 
and  points-of-contact.  The  computational  fluid 
dynamics  OKC  front  page  [Figure  19]  is  typical  of 
those  for  all  the  PET  functional  areas. 


Figure  19.  Sample  of  OKC  webpage 
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Table  4.  A  Sampling  of  Training  Courses  Given  in  FY  2006 


Course  Title 

ACES  III  Tutorial 
ABACUS® 

Advanced  EnSight 
AIM  Theory 
ANSYS® 

CFD  Case  Management  Workshop 
CFD  Days 

CFD  Pre-Processing,  Practice,  Current  Research,  and  Future  Directions 
CFD  Tools  and  Technologies 
Comprehensive  VHDL  Introduction 
Contact  in  LS-DYNA® 

Crosslight  APSYS  Training 
CUBIT 

Debugging  Parallel  Code  N/A  EMSF  Workshop 

FIELDVIEW 

FLUENT  Workshop 

Force  Field  Lectures 

HPC  FPGA  Programming 

Insight  Segmentation  and  Registration  Toolkit  Workshop  [ITK] 

Intermediate-Advanced  EnSight 

Introduction  to  Linux  Cluster  Computing:  Rocks  Cluster  Management:  and  Introduction  to  Parallel  Computing 

Introduction  to  HPC  Architectures  and  Parallel  Computing  and  Workshop  on  Program  Development  for 
Computational  Biology 

Introduction  to  Parallel  Programming  with  MPI 
Introductory  and  Advanced  MATLAB® 

Introductory  and  Intermediate  MATLAB® 

Introductory  EnSight 

Lattice  Heating  in  Quantum  Well  Laser  Diodes 
LS-DYNA®/ALE3D 

Mass  Conservation  Issues  in  Flow  and  Transport  Modeling 

PanIX:  Chemical  Driving  Force  Module  for  HPC  of  Materials  Behavior 

Parallel  GEMACS 

Parallel  Programming  Using  MatlabMPI 
Performance  Workshop  -  Bring  Your  Dwn  Code 
Python  for  Signal  Processing 
Scalable  Quantum  Chemistry  Applications 
Scientific  Computing  Using  Python  and  Perl 
Stars  3D 

SUGGAR  and  gViz  Training 


Number  of 
Attendees 

4 

17 
10 
8 
9 

9 

30 

36 

16 

10 

7 

8 

9 
21 
11 

14 
50 

10 

18 
10 
27 

15 

9 

20 

19 

16 

5 
11 
8 

20 
13 
30 
9 

39 

40 
15 
25 
19 
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FY  2006  Budget  Resources 

Financial  Analysis 

HPCMP  funds  are  used  for  [1 )  capitalization, 
sustainnnent,  and  operations  at  the  MSRCs;  [2] 
annual  capitalization  for  selected  ADCs  and  DHPIs; 
[3]  wide  area  network  services  for  the  DoD  HPC 
connnnunity;  [4]  investnnents  in  hunnan  capital  and 
key  HPC  software  applications;  and  [5]  expert 
HPC  services  from  leading  academic  institutions. 
Figure  20  displays  FY  2006  spending  by 
component  and  Figure  21 
shows  FY  2007  planned 
spending  by  component. 

We  use  multiple 
contracting  officers 
in  support  of  different 
efforts.  Contracting 
officers  at  the  General 
Services  Administration 
support  HPC  equipment 
and  services  purchases, 
and  contracting  officers  at 
various  DoD  installations 
support  our  service 
contracts.  This  structure 
is  necessary  because 
the  program  requires 
multiple  contracts  and 
contract  types  with  an 
ongoing  need  to  ensure 
that  state-of-the-art 
technical  capabilities  are 


made  available  to  DoD  scientists  and  engineers  in 
a  timely  manner.  Contracts  are  a  combination  of 
firm  fixed  price,  cost  and/or  indefinite  delivery/ 
indefinite  quantity.  All  procurement  awards 
are  made  for  commercially  available  systems. 
Acquisitions  are  accomplished  competitively  to  the 
fullest  extent  possible  and  encourage  the  inclusion 
of  small,  disadvantaged  businesses  and  MSIs. 


High  Performance  Computing  Modernization  Program 
FY  2006  Spending  by  Component 
[Percentage  of  Total  RDT&E  and  Procurement  Appropriated] 
(Including  All  Program  Assessments) 
$274,497,000 


Undistributed 
Congressional  RDT&E_ 
Reductions 
3.31% 


Undistributed 
Congressional 
Procurement  Reductions 


WHS  Management 
Reserve  (RDT&E) 

0.05% 

Final  RDT&E 

Withhold/Reprogramming, 
0.53% 


Congressional  RDT&E 
"Earmarks" 
10.92% 


Software  Initiatives 
19.84% 


Major  Shared  Resource 
Center  Operations 
27.19% 


Major  Shared  Resource 
Center  Upgrades 
14.95% 


Defense  Research  & 
Engineering  Network- 
11.96% 


Dedicated  HPC  Project 
Investments 
3.08% 


Allocated  Distributed 
Center  Operations 
6.71% 


Figure  20.  HPCMP  FY  2006  spending  by  component 
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High  Performance  Computing  Modernization  Program 
FY  2007  Planned  Spending  by  Component 
[Percentage  of  Total  RDT&E  and  Procurement  Appropriated] 
(Including  All  Program  Assessments) 
$260,980,000 


Undistributed 
Congressional 
Procurement  Reductions 
0.08% 


Undistributed 
Congressional  RDT&E 
Reductions 
2.16% 


Congressional  RDT&E 
"Earmarks" 
11.85% 


WHS  Management 
Reserve  (RDT&E) 
0.16% 


WHS  Management 
Reserve  (Procurement) 


Additional 

W  ithhold/Reprogram  m  ing 
[Projected] 

0.92% 


Software  Initiatives 
21.42% 


Major  Shared  Resource 
Center  Operations 
23.97% 


Major  Shared  Resource 
Center  Upgrades 
16.79% 


Defense  Research 
Engineering  Network- 
10.90% 


Dedicated  HPC  Project 
Investments 
2.40% 


Allocated  Distributed 
Center  Operations 
8.96% 


Figure  21.  HPCMP  FY  2007  planned  spending  by  component 


We  evaluate  the  effectiveness  of  each  progrann 
connponent  by  nneasuring  actual  cost  and  schedule 
perfornnance  versus  planned  cost  and  schedule 
performance  and  through  the  measurement 
of  actual  outcomes  versus  planned  outcomes. 
The  MSRC  contractors  submit  several  reports 
regularly  including  a  monthly  and  quarterly  cost 
performance  report  and  quarterly  contract 
funds  status  report.  Each  contract  specifies, 
as  a  deliverable,  a  work  breakdown  structure 
to  facilitate  the  on-going  review  of  smaller  task 
components.  Cost/schedule  status  reports 
are  one  of  the  primary  tools  used  for  oversight 
management  of  the  MSRCs. 

The  balance  sheet  on  page  58  shows  the 
cumulative  value  of  the  program. 


Obligations  and 
Costs 

□  ur  Financial 

Manager  conducts 
semi-annual  reviews  with 
each  major  component 
manager  and  major  field 
activity  to  review  actual 
cost  performance  against 
budgeted  cost  goals  in  a 
tailored  work  breakdown 
structure  format  with 
special  attention  on 
variance  analysis. 
Significant  variances  are 
reported  to  our  Program 
Director  and  corrective 
actions  taken.  We 
receive  approximately 
$250, each  year 
in  funding  appropriated 
for  the  DoD.  Cash 
flow  during  2006  is 
illustrated  by  the  Cash 
Flow  Statement  on  page  59. 

While  the  program  has  leveraged  major 
cost  performance  improvements  in  computer 
technology  since  1994,  validated  requirements 
have  always  exceeded  the  computing  capability 
available  to  address  those  requirements.  This 
occurs:  1]  because  the  use  of  science-based 
models  and  simulations  to  answer  research 
questions  and  solve  engineering  problems  has 
grown  dramatically;  and  2]  because  fully  funding 
the  HPC  requirement  is  unaffordable  given  the 
entire  scope  of  activities  the  DoD  budget  must 
address.  While  fiscal  resources  do  not  fully 
meet  the  computational  requirements  of  the 
science  and  technology  and  test  and  evaluation 
communities,  the  returns  provided  are  substantial 
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with  resources  allocated  to  the  highest  priority 

projects.  The  FY  2006  Income  Statement  on  High  Performance  Computing  Modernization  Program 

page  60  shows  these  shortfalls.  Acquisitions  by  Vendor 

(Percentage  of  Procurement  Appropriated) 
$53,500,000 

Financial  Trends 


Except  for  minimal  inflation  adjustments,  HPC 
budgets  are  essentially  flat.  We  address  urgent 
new  requirements  by  adjusting  priorities  within  the 
existing  funding  profile.  We  increased  the  overall 
capability  of  our  HPC  systems  by  about  80%,  and 
added  or  upgraded  systems  at  the  ADCs.  However, 
even  with  these  increased  capabilities,  we  are 
unable  to  meet  validated  DoD  requirements. 
Development  of  the  portfolios  and  institutes  will 
continue.  The  DoD  HPC  user  community  will 
continue  to  be  supported  by  the  PET  efforts. 
Our  Software  Protection  Initiative  will  continue 
to  mature.  Figure  20  breaks  out  program-wide 
and  planned  spending  during  2007. 


Figure  22.  HPCMP  FY  2006  acquisitions  by  vendor 


The  Income  Statement  on  page  60  shows 
that  currently  we  have  a  continuing  deficit.  The 
dollars  we  spend  are  not  keeping  up  with  the 
rapidly  growing  needs  of  the  scientific  community. 
Figure  22  displays  spending  by  vendor  in  FY  2006 
and  Figure  23  shows  planned  spending  by  vendor 
in  FY  2007. 


High  Performance  Computing  Modernization  Program 
FY  2007  Planned  Acquisitions  by  Vendor 
(Percentage  of  Procurement  Appropriated) 
$51,317,000 


Summary 

We  deploy,  sustain,  and  upgrade  commercially 
available  high  performance  computing 
environments  and  networking  services  in  support 
of  DoD  laboratories  and  test  facilities.  We 
have  substantially  improved  the  Department’s 
computational  capabilities  with  the  objective 
of  providing  the  DoD  the  technology  to  ensure 
dominance  on  the  battlefield  by  the  early  fielding 
of  the  most  advanced  computing  capability 
available. 


Figure  23.  HPCMP  FY  2007  acquisitions  by  vendor 
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High  Performance  Computing  Modernization  Program 
Balance  Sheet 
As  of  March  30,  2007 


Hardware 

Uncompleted  Software 

Less:  Depreciation 

$1,085,107,999 

Development 

$2,467,498 

Fiscal  Year  1994-2003 

$876,671,614 

$144,334,956 

Maintenance  Oontract  Liabilities 

Fiscal  Year  2004 

$34,101,429 

Hardware 

Fiscal  Year  2005 

$22,461,857 

Fiscal  Year  2007: 

$3,443,339 

Fiscal  Year  2006 

$7538,143 

Software 

Fiscal  Year  2007 

$0 

Fiscal  Year  2007: 

$471,449 

Software  [1 ) 

Intellectual/Facilities  Expense 

Less:  Depreciation 

$260,707356 

Government  Labor 

Fiscal  Year  1994-2001 

$130,715,508 

Fiscal  Year  2007: 

$5,355,228 

Fiscal  Year  2002 

$19,282,037 

Oontract  Labor 

Fiscal  Year  2003 

$13,789,787 

Fiscal  Year  2007: 

$18,326,818 

Fiscal  Year  2004 

$9,059,434 

Facilities 

Fiscal  Year  2005 

$6,828,937 

Fiscal  Year  2007: 

$12,171,628 

Fiscal  Year  2006 

$2,260,489 

Fiscal  Year  2007 

$0 

$78,771,164 

Manpower  Contracts  [2  &  3) 

Software  Development  Total  Liabilities  $42,235,960 


Exercised  Oontract  Value 

$23,204,057 

Less:  Value  Oonsumed  Remaining 
Exercised  Value 

$20,736,559 

$2,467,498 

Maintenance  Oontracts  [2  &  3) 
Hardware  Maintenance 

Fiscal  Year  2007: 

$14,506,216 

Software  Maintenance 

Fiscal  Year  2007: 

$1,937,007 

Less:  Value  Oonsumed 

Hardware  Maintenance 

Fiscal  Year  2007: 

$11,062,877 

Software  Maintenance 

Fiscal  Year  2007: 

$1,465,558 

$3,914,788 

Intellectual/Operations 

Government  Labor 

Fiscal  Year  2007: 

$23,835,734 

Oontract  Labor 

Fiscal  Year  2007: 

$91,821,495 

Less:  Value  Oonsumed 

Government  Labor 

Fiscal  Year  2007: 

$18,480,506 

Oontract  Labor 

Fiscal  Year  2007: 

$73,494,677 

$23,682,046 

Facilities 

Fiscal  Year  2007: 

$45,892,491 

Less:  Value  Oonsumed 

Fiscal  Year  2007: 

$33,720,863 

$12,171,628 

Program  Equity  $233,106,120 


Total  Assets  $265,342,080  Total  Liability  and  Program  Equity  $265,342,080 

[1 )  Research,  Development  and  Engineering  funding  used  to  develop  inventory  software. 

[2)  Office  of  Management  and  Budget  Circular  A-11,  Section  300  -  Planning,  Budgeting,  Acquisition,  and  Management  of  Capital  Assets  [Paragraph 
300.4),  defines  capital  assets  as  land,  structures,  equipment,  intellectual  property  [e.g.,  software),  and  information  technology  [including  IT  service 
contracts)  that  are  used  by  the  Federal  government  and  have  an  estimated  useful  life  of  two  years  or  more.  Therefore,  manpower  is  treated  as  a 
capital  asset. 

[3)  Small  consumable  items  such  as  computer  tapes  and  supplies  are  considered  as  expense  items  and  not  carried  as  inventory  items. 
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High  Performance  Computing  Modernization  Program 
Cash  Flow  Statement 
October  1,  2005  —  September  30,  2006 

Fiscal  Year  2006 


Revenue 

Research,  Development  and  Engineering  Funding 

President’s  Budget 

$189,747000 

Congressional  Funding 

$31,250,000 

Department  of  Defense  Reprogramming  -  In 

$0 

[Less  Congressional  Undistributed  Reductions] 

($9,077,000) 

[Less  Unreleased  Dbligation  Authority] 

($1,588,500) 

Net  Research,  Development  and  Engineering  Funding 

$210, 331, 500 

Procurement  Funding 

President’s  Budget 

$49,501,000 

Congressional  Funding 

$4,000,000 

Department  of  Defense  Reprogramming  -  In 

$0 

[Less  Department  of  Defense  Reprogramming  -  Dut] 

($734,000) 

Net  Procurement  Funding 

$52,767,000 

Net  Revenue 

$263,098,500 

Investments 

Major  Shared  Resource  Center  Upgrades 

$41,037722 

Allocated  Distributed  Center  Upgrades/Dedicated  HPC  Project  Investments 

$11,729,278 

Software  Development 

$21,474,645 

Expense 

Major  Shared  Resource  Center  Dperations 

$82,466,831 

Allocated  Distributed  Center  Dperations 

$40,567758 

Defense  Research  &  Engineering  Network 

$32,818,574 

Software  Initiatives 

$33,003,692 

Net  Expense 

$263,098,500 

Balance  [As  of  September  30,  2006) 

$0 
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High  Performance  Computing  Modernization  Program 


Income  Statement 

October  1,  2005  —  September  30,  2006 

Fiscal  Year  2006 

Income 

Research,  Development  and  Engineering  Funding 

Major  Shared  Resource  Center  Operations 

Allocated  Distributed  Center  Operations 

Defense  Research  &  Engineering  Network 

Software  Initiatives 

$82,466,831 

$40,567758 

$32,818,574 

$54,478,337 

Procurement  Funding 

Major  Shared  Resource  Center  Upgrades 

Allocated  Distributed  Center  Upgrades 

Defense  Research  &  Engineering  Network 

Software  Initiatives 

$41,037,722 

$11,729,278 

$0 

$0 

Total  Income 

$263, 098, 500 

Expense'’ 

Research,  Development  and  Engineering  Funding 

Major  Shared  Resource  Center  Operations 

Allocated  Distributed  Center  Operations 

Defense  Research  &  Engineering  Network 

Software  Initiatives^ 

$82,466,831 

$40,567758 

$32,818,574 

$33,033,692 

Depreciation  of  Capital  Assets 

Hardware  [Depreciated  based  upon  a  48-month  life-cycle]^ 

Software  [Depreciated  based  upon  a  6C-month  life-cycle]"^ 

$55,384,000 

$20,066,079 

Total  Expense^ 

$264,306,934 

Balance  [As  of  September  30,  2006) 

($1,208,434) 

Note  1 :  Expenses  include  travel;  supplies:  government  and  contractor  salaries  and  training:  maintenance  of  hardware  and  software: 
studies  and  analysis:  annual  operations  investments:  communications,  utilities,  facilities  lease,  and  facilities  maintenance. 

Note  2:  Software  initiatives  are  separated  into  two  distinct  categories— expenses  associated  with  research  and  development,  management, 
education/training,  and  expert  services:  and  capitol  assets  resulting  from  developed  software. 

Note  3:  Depreciation  for  HPC  hardware  is  calculated  using  a  48-month  straight-line  depreciation  method.  Current  HPC  technology 
development  results  in  predictable  obsolescence.  Generally  after  48  months  of  use,  HPC  systems  are  retired  with  little  or  no  residual 
value.  Fiscal  year  20C6  depreciation  includes  the  12-month  value  calculated  for  all  systems  in  the  inventory  between  October  1,  2005 
through  September  30,  2006. 

Note  4:  Depreciation  for  HPC  software  is  calculated  using  a  60-month  straight-line  depreciation  method.  A  period  of  60  months  is  used 
because  it  is  the  typical  life  cycle  of  HPC  software  before  significant  modifications  are  required.  Fiscal  year  20C6  depreciation  includes 
the  12-month  value  calculated  for  all  software  in  the  inventory  between  October  1,  2005  through  September  30,  2006. 

Note  5:  Annual  program  investments  in  system  hardware  have  not  been  made  at  levels  sufficient  to  maintain  stable  equipment  inventories. 
For  several  years  depreciated  values  have  not  been  offset  by  new  assets. 
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Acronyms 


Acronyms 


2- D  two-dimensional 

3- D  three-dimensional 

ADCs  allocated  distributed  centers 

AFB  Air  Force  Base 

AFRL/IF  Air  Force  Research  Laboratory,  Information  Directorate 

AFRL/SNHE  Air  Force  Research  Laboratory,  Sensors  Directorate,  Electromagnetic  Scattering 
Branch 

AHPCRC  Army  High  Performance  Computing  Research  Center 

ARL  Army  Research  Laboratory 

ARSC  Arctic  Region  Supercomputing  Center 

ASC  Aeronautical  Systems  Center 

ASK  Armor  Survivability  Kit 

BEI  Battlespace  Environments  Institute 

BHSAI  Biotechnology  HPC  Software  Applications  Institute  for  Force  Health  Protection 

C4ISR  command,  control,  communications,  computers,  intelligence,  surveillance  and 

reconnaissance 

CAP  capability  applications  projects 

CBoD  Centers  Board  of  Directors 

CCM  computational  chemistry,  biology,  and  materials  science 

CDLT  collaborative  and  distance  learning  technologies 

CE  computational  environment 

CEA  computation  electromagnetics  and  acoustics 

CERDEC  Command  Electronics  Research,  Development  and  Engineering  Center 
CFD  computational  fluid  dynamics 

CPU  central  processing  unit 

CREATE  Computational  Research  and  Engineering  Acquisition  Tools  and  Environments 

CSM  computational  structural  mechanics 

CST  collaborative  simulation  and  testing 

CTAs  computational  technology  areas 

CWO  climate/weather/ocean  modeling  and  simulation 

DARPA  Defense  Advanced  Research  Projects  Agency 

DHPIs  Dedicated  HPC  Project  Investments 

DIOT  Distributed  Implementation  and  Operations  Team 
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DoD 

Department  of  Defense 

DOE 

Department  of  Energy 

□REN 

Defense  Research  and  Engineering  Network 

DPG 

Dugway  Proving  Ground 

ED&PMB 

Engineering  Design  and  Process  Management  Board 

ESMF 

Earth  Systems  Modeling  Framework 

ENS 

electronics,  networking,  and  systems/C4l 

EOTC 

education,  outreach,  and  training  coordination 

ERDC 

Engineer  Research  and  Development  Center  [USAGE] 

EQM 

environmental  quality  modeling  and  simulation 

ET 

enabling  technologies 

FLOPS 

Floating-point  Operations  per  Second 

FMS 

forces  modeling  and  simulation 

FY 

fiscal  year 

GAO 

Government  Accountability  Office 

GFs 

gigaFLOPS 

GMTI 

ground  moving-target  indication 

HI-ARMS 

HPC  Institute  for  Advanced  Rotorcraft  Modeling  and  Simulation 

HPC 

high  performance  computing  or  high  performance  computer 

HPCMP 

High  Performance  Computing  Modernization  Program 

HPCS 

DARPAs  High  Productivity  Computing  Systems 

lED 

Improvised  explosive  device 

IHAAA 

Institute  for  HPC  Applications  to  Air  Armament 

IM 

Insensitive  Munitions 

IMT 

integrated  modeling  and  test  environments 

IMTPS 

Institute  for  Maneuverability  and  Terrain  Physics  Simulation 

ISSA 

HPC  Software  Applications  Institute  for  Space  Situational  Awareness 

JFCOM 

Joint  Forces  Command 

JET 

Joint  Engineering  Team 

LSN 

large  scale  network 

MDA 

Missile  Defense  Agency 

MFT 

multiphase  flow  target  response 

MHPCC 

Maui  High  Performance  Computing  Center 

MiniSAR 

Miniaturized  Synthetic  Aperture  Radar 

MIT 

Massachusetts  Institute  of  Technology 

MSIs 

Minority  Serving  Institutions 

MSRCs 

major  shared  resource  centers 
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NASA  National  Aeronautics  and  Space  Administration 

NAVO  Naval  Oceanographic  Office 

NUWC  Naval  Undersea  Warfare  Center 

OKC  Online  Knowledge  Center 

OneSAF  One  Semi-Automated  Forces 


OOS 

OSD 

PCID 

PET 

PEUO 

RDT&E 

ROI 

S&T 

SAR 

SAS 

SIP 

SMDC 

SSA 

SSCSD 

SSN 

STAP 

T&E 

Tl 

US 

USACE 

VED 

WAN 


One  Semi-Automated  Forces  (OneSAF)  Objective  System  (DOS) 

Office  of  Secretary  of  Defense 

physically-constrained  iterative  deconvolution 

User  Productivity  Enhancement  and  Technology  Transfer 

physics-based  environment  for  urban  operations 

research,  development,  test,  and  evaluation 

r  etu  r  n-o  n-i  nvestm  ent 

science  and  technology 

synthetic  aperture  radar 

Software  Applications  Support 

signal/image  processing 

Army  Space  and  Missile  Defense  Command 

space  situational  awareness 

Space  and  Naval  Warfare  Systems  Center,  San  Diego 

Space  Surveillance  Network 

space-time  adaptive  processing 

test  and  evaluation 

technology  insertion 

United  States 

US  Army  Corps  of  Engineers 
virtual  electromagentics  design 
wide  area  network 
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